Developmental learning of linguistic structures through limited experience of visuo-proprioceptive streams in a humanoid robot

Abstract

This study investigates developmental learning of linguistic concept acquisition associated with visuo-proprioceptive experience using a simulated humanoid robot. We proposed novel dynamic deep neural network model for processing multimodal information stream. The proposed model is composed of dedicated submodules for visual perception, behavior generation, language comprehension, and integration of those different modalities. The simulated robot learns to associate a set of imperative linguistic phrases with raw visual inputs from the simulator and object-directed actions through two learning stages from human designed curriculum. The generalization capability in the second stage can be achieved from limited number of additional exemplars by utilizing the prior knowledge of the first stage. The experimental result showed that the proposed model can generalize the learned skills of action generation to unfamiliar object configurations. We also examined whether the robot acquired linguistic concept of action and object from limited experiences by visualizing the activity of neurons using a dimensionality reduction technique. The analysis showed that the robot could acquire structured representation of the language associated with the visual features of the object and the dynamical properties of the action.