R-LSTM

Abstract

In Natural Language Understanding (NLU), sequence models such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) are widely used because they can learn phrase and sentence representations that are sensitive to word order. However, the sequence models share a common limitation that previous contextual information tends to be forgotten as new words are processed. It makes the sequence models hard to learn the relations among new word and words processed in the previous. To address the limitation, we propose a sequence model called Restorable Long Short-Term Memory (R-LSTM) that can restore the forgotten information needed to understand the current word. R-LSTM is an extension of the LSTM by applying two approaches: a multiple attention mechanism with coverage loss and a restore gate. The multiple attention mechanism with coverage loss extracts the relevant contexts from the stored previous contexts. The relevant contexts are selectively reflected into current memory of R-LSTM through the restore gate. In other words, the restore gate enables our R-LSTM to restore only the forgotten and meaningful information. We evaluate our model on various NLU tasks and data sets: language modeling (WikiText-2 data set)

natural language inference (SNLI data set)

text classification (AG's News, Yahoo! Answers, Yelp Review Polarity data sets). In general, experimental results on various tasks show that the performance of our model outperformed in comparison of conventional sequence models. Moreover, the exploratory experiments on WikiText-2 data set are conducted to assess that each proposed approach affects positively on the performance.