Improved recurrent neural networks for grammatical inference

Abstract

Various artificial network models have been investigated for the problem of deriving grammar rules from a finite set of input examples. Among them, discrete recurrent network are more robust and more stable than any other network models. But due to gradient descent mechanism, even discrete recurrent network is not powerful enough for a certain input examples. In this thesis, in order to improve such recurrent networks with gradient descent rule we first present the modified learning algorithm associated with them. Then, we propose the modified second-order recurrent networks model with the structural clustering learning method in order to make the model more powerful for any type of input examples. Such two methodologies are presented and studied in learning context-free grammars through second-order recurrent networks with a external stack. Hence, we present three main issues as follows;

● the improved second-order recurrent neural networks,

● the equivalent second-order recurrent neural networks(ESRNN), and

● the ESRNN for grammatical inference

In the improved second-order recurrent networks, we propose two considerations that improve the learning performance of the existing second-order recurrent neural networks for inferring regular grammars. One is to arrange input strings in well-ordered form and use them as inputs. Experiments were performed to evaluate the learning performance according to input order types and decide the best type. The other is to analyze the cause of the failure to learn due to the recursive gradient-descent rule and modify the algorithm so as to reduce total learning time. Using such considerations, we suggest the modified learning method and show the enhanced result of experiments. In the equivalent second-order recurrent networks, second-order recurrent neural networks including discrete recurrent networks and analog recurrent network have such problems as the restriction of the number of training examples and the restriction o...