Human-Robot full-sentence VQA interaction system with highway memory network

Abstract

One of the major functions of intelligent robots such as social or home service robots is to interact with users in natural language. Moving on from simple conversation or retrieval of data stored in computer memory, we present a new Human-Robot Interaction (HRI) system which can understand and reason over environment around the user and provide information about it in a natural language. For its intelligent interaction, we developed a deep learning network model based on Dynamic Memory Networks (DMN), a deep learning network for Visual Question Answering (VQA), and proposed Highway Memory Network (HMN) and Full-Sentence Highway Memory Network (FSHMN). For its hardware, we built a robotic head platform with a tablet PC and a 3 DOF neck. Through an experiment where the user and the robot had question answering interaction in our customized environment and in real time, the feasibility our proposed system was validated, and the eectiveness of deep learning application in real world as well as a new insight on human robot interaction was demonstrated.