(An) expert data-driven self-supervised learning of air combat maneuver model

Abstract

This thesis address a model learning problem that imitates air combat maneuver given expert trajectory data. To learn an imitation model from given data, the challenges of constructing models to represent air combat maneuver and control techniques need to be taken into consideration. The World Models, one of the approaches of self-supervised learning, is adopted to model the unknown system and to learn controller from the data. The World Models is a probabilistic representation learning framework that represents the internal model as a mixture of Gaussian Recurrent Neural Network using time series images given in a real environment and enables system model learning using time series data and controller learning through hidden states. While the controller trained based on reinforcement learning, a reward function for state-action pair is required. Such a reinforcement learning method is difficult to apply to our problem where the reward function is unknown. To overcome this difficulty, a self-supervised learning approach termed Imitative World Models(IWM) is proposed that incorporates World Models and imitation learning methods. The proposed framework learns the internal model to represent air combat and also learns expert-like policies without a reward function. The controller of the proposed model enables predictive control by receiving future prediction information from the internal model as well as the current states. Furthermore, a dreaming process makes it possible to create trajectories similar to expert data and take challenging actions without any interaction with the real environment. The proposed model-based imitation learning framework cannot generate an appropriate maneuver due to the uncertainty of the model when a large deviation occurs between the training and test data. In this paper, this distribution mismatch problem is tackled by measuring the degree of uncertainty using a mixed Gaussian network and modifying the control value with that of another controller. Numerical experiments show the similarity of reconstructed trajectories from the proposed method and expert trajectories. The performance of the IWM is verified through comparison with existing model-free imitation learning.