Multi-agent deep reinforcement learning in dynamic, adversarial environment

Abstract

In this work, the capability of deep Q-network, a type of deep reinforcement learning algorithm, is examined on a dynamic, multi-agent environment AI soccer simulation game. In the AI soccer simulation game, two teams of three differential-wheel robots compete as in the real soccer game, pushing the orange-colored ball into each other’s goal area to earn more score than the opponent team. The simulation game provides various data including the top-view image of the soccer field, positions and orientations of the robots and the ball, scores, etc. to each team’s controller in every simulation step to be used as the sources for learning and playing the AI soccer game. To control three robots belonging to the home team, two or three deep Q-networks are trained on the AI soccer environment. One deep Q-network is assigned to control a goalkeeper robot. The other two robots are the attackers and controlled in two ways. In one method, one deep Q-network controls two robots simultaneously. In the other method, two deep Q-networks control two robots separately. The deep Q-networks take the top-view image of the soccer field as the input and output the ID of primitive action to be executed by the robot they control. The rewards are set as to motivate the robots to take the role of a goalkeeper and two attackers. For training the deep Q-networks, different sessions are held to train the goalkeeper and two attackers separately and then simultaneously. Through evaluation of the training sessions, the possibility for the deep Q-network to learn how to play the AI soccer game when adequate state, actions, and rewards are defined is shown.