Constrained bayesian reinforcement learning via approximate linear programming

Abstract

In many situations, too much exploratory behaviours can cause severe damage to the reinforcement learning agent and there should be restrictions on such behaviours. These restrictions can naturally be encoded as CMDPs where cost functions and cost constraints represent the risk of behaviours and the degree of risk taking respectively. We propose model-based Bayesian reinforcement learning (BRL) algorithm in CMDP environment, showing risk-sensitive exploration in a principled way. Our algorithm efficiently solve the given constrained BRL problem through finite approximation of the original belief-state CMDP's linear program, and generates a finite state controller in an off-line manner. We provide the corresponding theoretical guarantees and empirical supports that the proposed method outperforms the previous state-of-the-art approach.