Learning to factorize with regularization for cooperative multi-agent reinforcement learning

Abstract

Multi-agent reinforcement learning tasks require that agents learn in a stable and scalable manner. To this end, we explore solutions in centralized training and decentralized execution (CTDE) regime popularized recently and focus on value-based methods. VDN and QMIX are representative examples employing centralized training to resolve instability and non-stationarity issues, and distributed execution to render the algorithm scalable. While appropriately factorizing the joint value functions into individual ones is key to distributed execution, we find that the existing methods of value function factorization address only a fraction of game-theoretically modelable MARL tasks. We propose QREG, which takes on a new approach to value function factorization: regularizing the joint value function. This approach translates to relaxing the previously assumed conditions placed on the nature of the value functions. Upon relaxing those assumptions, we illustrate that QREG covers every game satisfying a set of relatively mild conditions, enabling QREG to cover a wider class of games. Our simulations indicate superior performance in a variety of settings, with especially larger margins in games whose payoffs penalize non-cooperative behavior more harshly.