Lane-Merging of autonomous vehicle via finite state machine and policy-based RL

Abstract

In order for autonomous vehicles to reach their desired destinations, it is important to perform lane changes safely and quickly when it is needed. However, lane change, unlike lane keeping, has increased interaction with surrounding vehicles, making it difficult to predict future risk of ego vehicle. Especially in the lane merge section, the risk is maximized due to simultaneous lane changes. In this study, we propose a methodology that shows high cut-in success rate and collision avoidance rate in various lane merge scenarios. Conventional lane change algorithms are based on rules with physical meanings, but the simplicity of the logic makes it difficult to cope with a myriad of lane change situations. In addition, a lane change algorithm using a model such as MPC has limitations in a complicated situation such as a merge section in which it is difficult to express road conditions as a probabilistic model. In order to overcome this problem, deep reinforcement learning based lane change research has been actively conducted in recent years. However, due to the lack of interpretability, which is a limitation of end-to-end deep learning methodology, there is a difficulty in applying a lane merge scenario in which the risk of ego vehicle is maximized. To overcome this problem, this paper proposes a methodology for performing low-level cut-in through reinforcement learning after high-level decision making using rule-based finite state machines. First, in the high-level decision step, the target driving state (ready / approach / negotiate / lane change / finish) of ego vehicle is determined. The target-driving mode is determined in real time through an algorithm that predicts the risk of surrounding vehicles and selects an optimal target gap. After the target driving state is determined as lane change, Soft actor-critic based cut-in algorithm is executed, which is suitable algorithm for complex scenario by encouraging exploration of ego agent. In order to secure safety and high cut-in success rate, inputs and reward function of RL network is constructed to perform lane change with securing minimum safety distance. In addition, we trained lane change strategies with the traffic simulator that solved the long-tail problem of data so that the strategy could be fully learned in various lane change scenarios including dangerous situations. By incorporating a rule-based algorithm into the deep reinforcement learning methodology, we developed an interpretable algorithm that can adjust the trade off between lane change success rate and collision avoidance rate. When the proposed algorithm is compared with the existing rule-based algorithm, the performance of collision avoidance and lane change success rate is improved in various traffic scenarios. In particular, it was found that the lane change success rate increased by 22% compared to the rule-based methodology under heavy traffic.