Sequential decision making with only return and action

Abstract

As recent success of transformer architectures have shown superior performance in sequence modeling, several approaches have been proposed to apply transformers in various fields, including sequential decision-making and reinforcement learning, such as the prior work on Decision Transformers. However, Markov Decision Processes (MDPs), the standard problem setting in sequential decision making and reinforcement learning, require information on the transition sequence of state, action, and reward. This information is not always available in real-world problems. In this paper, we propose a new problem setting for decision making, which is a relaxation of the MDP that requires fewer conditions, thus making it easier to apply in many real-world situations, such as robotic control or experimental design. By extending the approach used in Decision Transformers, we suggest a decision making method that leverages the sequence modeling power of transformers in this new problem setting. Additionally, we propose an active learning framework that could enable goal-oriented active learning in this new problem setting, using uncertainty modeling and sequence generation.