Offline-to-online reinforcement learning via balanced experience replay and pessimistic Q-ensemble

Abstract

Recent progress in offline reinforcement learning (RL) has made it possible to train strong RL agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such offline RL agents via further online interaction. Here, we make an observation that state-action distribution shift may lead to severe bootstrap error during fine-tuning. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple pessimistic offline Q-functions, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method stabilizes Q-learning during fine-tuning and improves the final performance and sample-efficiency of fine-tuned agents on various continuous control tasks from the D4RL benchmark suite.