Multiple UAV routing for re-planning under dynamic environment using deep reinforcement learning

Abstract

This thesis studies an autonomous mission planning algorithm that enables unmanned aerial vehicles (UAVs) to autonomously re-plan missions without human direct intervention under various dynamic mission environment changes, such as the creation of new missions, cancellation or change of existing missions. In this study, the mission re-planning problem is developed based on the Team Orienteering Problem (TOP), one of the variants of the Vehicle Routing Problem with Profits (VRPP). Existing deep reinforcement learning (DRL) based methods have largely focused on learning heuristics for Vehicle Routing Problem (VRP) and its variants that intrinsically have vehicles departing from a given depot and returning to that depot. While this setting is necessary to plan routing missions in advance, it needs to be expanded to cope with mission re-planning scenarios where vehicles are located away from the depot at the start. Additionally, many real-life re-planning situations are subject to a fuel constraint on each vehicle, which is likely to have variable remaining fuel. Therefore, this thesis investigates a Multiple-Start TOP (MSTOP), in which vehicles begin at multiple random locations, travel to maximize the total prizes, and arrive at the given depot, while satisfying fuel constraints. To solve MSTOP, this thesis proposes a methodology consisting of self-attention mechanism on each partial tour, and encoder-decoder attention mechanism between partial tour and remaining nodes. The proposed DRL-based method produces a suboptimal solution comparable to the existing meta-heuristic techniques, even for more complex problems. Furthermore, several case studies are presented to demonstrate the performance of the proposed model and solution procedure.