Reinforcement learning based joint task allocation and waypoint selection with robotic agent system design

Abstract

Vehicle routing problems (VRPs) are well-known NP-hard problems. Although mathematical formulations for VRPs are intuitive and straightforward, it requires a lot of computational resources to be solved. There have been twofold approaches to address contexts of VRPs: 1) Exact solution approaches and 2) Heuristic approaches. Recently, there has been much interest in applying machine learning-based solver as heuristic-based approaches.

In this dissertation, we consider the system of capacitated vehicles. That is, the system consists of multiple vehicles and infrastructures for any vehicle operations, such as multiple depots and replenishing stations. Because vehicles in the real world have limited capacities, which have critical effects on the system's performance, therefore, the system's purpose can be achieved by wise strategies for vehicle moves and deployment of the infrastructures. However, both vehicle moves and deployment of the infrastructures have a strong correlation, so that we solve both problems jointly.

Generally, in this dissertation, we develop a framework consisting of a simulation (environment) called Simulator, and a learning agent called Learner. We model the target problems mathematically in a manner of an MDP formulation to develop the Simulator for state-based decision making. We also develop the Learner based on DRL (Deep reinforcement learning) and implemented several DRL approaches and sub-algorithms for the robustness and the scalability of the learning process. At last, we avoid an instance learning where the learning process should be repeated in all different problem instances by acquiring transferability by transfer learning.

We propose several RL-based frameworks such as 1) DQN, 2) A2C with temporal abstraction, and 3) MARL with multiple GNNs to solve the mCVRP and system design problem. Target problems are CVRPs (Capacitated vehicle routing problems) and system design problems where CVRPs dictate the routing plans for the multiple capacitated vehicles, and system design problems determine the optimal deployment of the infrastructures in given target problems.

First, the DQN-based framework is studied to show both the robotic agents' movement and system design can be jointly solved in the context of reinforcement learning. However, the MDP model and the DQN-based \textit{Learner} can solve small size problems only due to the nature of the value-based RL and the single-agent RL. Second, we use the framework of A2C with temporal abstraction to achieve scalability. The temporal abstraction can split the whole network into a vehicle routing part and a system design part. As a result, both problems can be split mathematically, resulting in a hierarchical RL framework.

Based on the split framework, we concentrate on the vehicle routing problems only. We use an MARL (Multi-agent reinforcement learning) to efficiently address the vehicle routing problems. We also exploit multi-GNNs (multi-graph neural networks), representing a current state as embedding vectors, to achieve transferability and scalability. The framework of MARL with multi-GNN alleviates the curse of dimension by controlling each vehicle using an individual actor. Further, a trained GNN can extract latent factors for a current state, which is an achievement of generality for any state spaces.