Deep reinforcement learning-based intelligent agent for autonomous air combat

Abstract

The purpose of this study is to make the AI model perform autonomous combat by itself, focusing on the close air combat problem, especially in the air combat situation. There will be more diverse engagement situations in the future aerial engagement, and it is difficult to solve all these unpredictable engagement situations with the existing rule-based method. Therefore, in this study, using deep reinforcement learning, we developed an intelligent agent that can make self-judgment and act for combat maneuvers in close air combat as well as autonomous flight. When the agent gets input data in a fighting situation, state representation is performed for more efficient feature extraction. The agent's behavior is determined by the output action in the continuous action space just like the real fighter pilots control it. In addition, the reward function, which has the most influence on the agent's self-judgment of the air combat situation and selection of the optimal action, was appropriately designed. The transition corresponding to the agent's state change was performed with a transition representation to reflect the delayed responsiveness of the control stick. As a reinforcement learning algorithm, the representative Proximal Policy Optimization (PPO) in the on-policy technique and the representative soft actor-critic (SAC) in the off-policy technique were used. By comparing the two algorithms, SAC algorithm that shows a little faster and more stable learning performance was finally selected. We used a digital combat simulator as an environment for training agents. The intelligence model trained with the proposed various representation technique reinforcement learning algorithm is verified through testing. To confirm the robustness of the model, verification was performed with a new mission, and the possibility of expanding the trained model was confirmed using various types of enemy fighters.