Deep neural networks incorporating discourse information for modeling text

Abstract

Capturing semantics scattered across entire text is one of the important issues for NLP tasks. It would be particularly critical with long text embodying a flow of themes. This paper proposes a new text modeling method that can handle thematic flows of text with Deep Neural Networks (DNN) in such a way that discourse information and distributed representations of text are incorporate. Unlike previous DNN-based document models, the proposed model enables discourse-aware analysis of text and composition of sentence-level distributed representations guided by the discourse structure. More specifically, my method identifies Elementary Discourse Units (EDUs) and their discourse relations in a given document by applying Rhetorical Structure Theory (RST)-based discourse analysis. The result is fed into a tree-structured neural network that reflects the discourse information including the structure of the document and the discourse roles and relation types. I evaluate the document model for two document-level text classification tasks, sentiment analysis and sarcasm detection, with comparisons against the reference systems that also utilize discourse information. In addition, I conduct additional experiments to evaluate the impact of neural network types and adopted discourse factors on modeling documents vis-à-vis the two classification tasks. Furthermore, I investigate the effects of various learning methods, input units on the quality of the proposed discourse-aware document model.