Conjoined architecture for heterogeneous multi-agent reinforcement learning

Abstract

In this dissertation, we propose the conjoined architecture designed for multi-agent reinforcement learning in environments using global state and heterogeneous agents. Under such environments, semantic mismatches in both input and output layers arise and hinder contemporary multi-agent reinforcement learning algorithms to efficiently train under centralized-training decentralized-execution settings using parameter sharing. On that regard, we propose the conjoined architecture capable of effectively train in environments using global state and heterogeneous agents. The conjoined architecture is a partially parameter sharing architecture where heterogeneous agents are considered as a single team to be trained together using global state as the input to be processed through a team network. Unlike traditional fully-centralized training, the conjoined architecture factorizes the output joint action space into individual agents' action spaces represented with agents' own weights and biases. We exemplify the use of the conjoined architecture through proposing two actor-critic algorithms multi-actor-conjoined-critic and conjoined-actor-conjoined-critic. A conjoined critic evaluates all agents' actions as a single sample. Instead of evaluating joint action-space values for all action combinations of agents, the conjoined critic outputs individual Q-values for each agent to reduce output dimension size. Through value decomposition network, individual Q-values are summed to estimate team Q-values, which is the optimization objective for the critic. For multi-actor-conjoined-critic, individual actors are trained with value estimations from conjoined critic while sharing their internal state among each other through bandwidth-limited communication channel. For conjoined-actor-conjoined-critic, a parameter-efficient conjoined actor is used in addition to the conjoined critic to replace individual actors. We evaluate the proposed algorithms in AI Soccer environment that uses global state and heterogeneous agents and compare the results with existing algorithms to demonstrate conjoined architecture's effectiveness. Finally, we conduct ablation studies to investigate effects of components in the proposed algorithms.