Models and algorithms for inverse reinforcement learning

Abstract

Reinforcement learning (RL) is the problem of how an agent can learn to behave optimally. A reward function plays an important role in determining the optimality because it specifies how much reward or punishment is given in every situation. We thus need an appropriate reward function that describes the objective of the problem or the preference of the agent to formalize an RL problem. However, it is a difficult task in practice. The reward function is often hand-tuned by domain experts iteratively until a satisfactory strategy is obtained via RL algorithms. Therefore, a systematic way to determine the reward function is highly desired to avoid this labor-intensive process. The main focus of this thesis is inverse reinforcement learning (IRL), which aims to infer the reward function that the domain expert is optimizing from her behavior data. Since IRL provides a framework to explore the principle of the behavior, it can be utilized in various research areas such as examining human and animal behaviors, building intelligent agents that imitate the demonstrator, and developing an econometric model for making decisions. A number of studies on IRL algorithms have appeared in the literature during the last decade, but there remain a number of challenges: (1) The IRL problem is inherently ill-posed. There are infinitely many reward functions that make the expert’s behavior optimal. (2) The expert is generally assumed to behave optimally, but she may choose sub-optimal actions. (3) The behavior data is typically assumed to be generated by a single expert having a single reward function. In practice, it is often gathered from a number of experts to obtain the enough amount of the data. (4) When dealing with large problems, we assume that the pre-defined features are given and find the reward function as a linear function of the features. However, it is difficult to specify the features that compactly represent the reward structure. (5) Although the expert is genera...