Feature learning method for grayscale images based on bidirectional associative memory

Abstract

Feature learning has attracted much research concern for years as an important component of machine learning, pattern recognition and image processing. Its goal aims at finding a way to extract and understand features effectively from abstract and complex patterns. In this field of application, many techniques has been employed, namely Principal Component Analysis (PCA), Auto-encoder (AE) or Restricted Boltzmann Machine (RBM). In our thesis, we focus on employing Bidirectional Associative Memory (BAM), a type of Recurrent Neural Network, to learn the features. The original idea of this model was proposed by Kosko in 1988. In his work, Kosko introduced the two-layer nonlinear feedback neural networks of which stability and encoding properties are demonstrated. The stable state of BAM corresponds to a system local minimum. Given two input patterns, BAM tends to memorize them by gradually seeping pattern information into the synaptic weights W, following Hebbian learning law. By the same way, our extraction method looks forwards to store each input pattern and its extracted feature in the local minimum of the network state space. To do that, we combine BAM model and architecture with Energy-based learning approach that is proposed by Yann LeCun in 2006. Several trials has been done to achieve the goal. In the beginning, we tried to simply minimize Energy function of the pairs “input pattern - extracted feature”, along with adding several regulation terms to restrict the model. However, we found this idea may result in saturation state in which different patterns recall the same feature. Another way is to minimize energy gap between the input pattern and its first retrieval one. This method tends to push down on the energy of our input pattern, while it will pull up on wrong energy that is assumed to be the first retrieval. In experiments, we test our proposed method on grayscale images set MNIST. For evaluation, BAM was compared to RBM, AE and its variants. Anoth...