Two mode Q-learning using failure experience of the agent and its application to biped robot

Abstract

In this thesis, a two mode Q-learning method is proposed for fast convergence, extending Q-learning, a well-known scheme in reinforcement learning. It employs a separate failure Q value that keeps track of failure experiences and uses this to modify the exploratory behavior of a learning agent. The effectiveness of the pro-posed two mode Q-learning method is verified in a grid world environment. Its performance is also evaluated against conventional Q-learning in training a soccer agent to perform goalkeeping. The acquired knowledge of two mode Q-learning is implemented on the goalie robot of the NaroSot soccer system. The effects of varying parameters in two mode Q-learning is investigated. Also, a biped robot, called HSR-IV, is used as a test bed to compare the performance of both algorithms. An external force that is generated in the sagittal plane, is applied to the HSR-IV and its standing posture is investigated. Q and two mode Q-learning are employed to select an action for resisting the external force. In the frontal plane, an external force is generated and impacts the HSR-IV. In this situation, more than two actuators that were considered in this thesis, exist for resisting the external force. For implementing Q and two mode Q-learning in this situation, a curse of dimensionality must be considered. To solve this problem, a module-based scheme is adopted. The effectiveness of module-based two mode Q-learning is verified by real experiments using HSR-IV.