Efficient and interpretable recurrent neural network grammars

Abstract

In this work, we are motivated by the hypothesis, that a linguistically informed inductive bias is beneficial to NLP models. Based on this hypothesis, we seek to improve a model with such an inductive bias: Recurrent neural network grammars (RNNG). Our primary contribution lies in Parallel RNNG, where we obtain 10 - 15 times speedups for large batch sizes. We also propose to improve RNNG with the use of attention. This makes it possible to interpret RNNG, but causes the predictive power to slightly deteriorate. Finally, we apply well-known regularisation methods from RNN to RNNG and show that they improve the predictive power of RNNG. All of our methods are evaluated against English and Chinese treebanks and compared to baselines without [strong] inductive biases.