Development of traffic signal control model using reinforcement learning

Abstract

Recently, with the development of various traffic information collecting system, high-quality traffic big data has been collected and used for the traffic operation. However, the current traffic signal control system has low data availability so that its efficiency becomes less effective when the traffic demand is oversaturated, since it still uses the outdated model-driven methods. Accordingly, two alternative approaches are suggested to develop a next-generation signal control system: smart intersection signal control method using artificial intelligence (AI), and perimeter control for managing network-level traffic demand. These two approaches are state-of-the-art signal control methods proposed in the last decade, and various strategies for expanding to a large-scale urban area have been proposed in recent few years. However, it is cost-expensive to apply each method for the signal control optimization of a large-scale urban area. Accordingly, it is more effective to integrate both signal control methods rather than to apply a single method. As the first step for the integration of these two signal control methods, this dissertation explores optimization strategies for the two signal control methods. In particular, we utilize reinforcement learning (RL) as a key methodology with the consideration of traffic big data availability, improvement of uncertainty and computational cost of the conventional model-based approaches, and the scalability to large areas. To this end, this dissertation aims to apply RL for the signal control of (1) isolated single intersection, (2) multiple intersections of urban district, and (3) perimeter control of large-scale urban networks. This dissertation provides a basic study to introduce RL to the traffic signal control problems by discovering some related issues, proposing some solutions to overcome the issues, and validating them with the simulation experiments. Regarding the development of AI-based smart intersection signal control, this dissertation aims to obtain the following three goals. The first is an effective variable design for signal controls of large-scale urban areas. Although there are a number of signal control variables, most of current studies do not consider the relationship between the variables rather they suggest some infeasible and inefficient signal control designs. Therefore, we initially classify the key variables and analyze the conventional signal control methods. In addition, we employ the phase duration split as the main signal control variable while the other variables are fixed. The optimization of signal cycle and offset or the optimal design of signal phase and rings are left for the future studies. Second, we identify an issue of restricted exploration which has a great influence on the robustness of the model. The restricted exploration problem is caused by the travel demand scenario of traffic simulation, which severely degrades the model’s efficiency in some unexperienced traffic states. Thus, we employ a graph representation for the traffic state of the intersection. The traffic state is expressed as graph-structured data, and it is trained using a graph neural network (GNN) to represent the traffic state in a generalized topological space. This method enables to obtain a transferable policy that can adapt to a new unexplored state by utilizing already-trained knowledge of topologically equivalent states. The proposed method is validated with a few unexplored travel demand scenarios of an isolated single intersection under well-controlled experimental conditions. The results show that the model using a graph-representation has improved the transferability of the policy compared to the model without applying the graph. Third is the design of an effective method for training multi-agent signal control agents. For the coordination of multiple intersections, it requires to obtain a cooperative solution considering the interactions between them. However, most current studies only observe information of neighboring intersections without considering the spatial correlation between the intersections. Hence, this dissertation determines the structure of the graph and extracts the features defining the variables based on the cell transmission model (CTM), the basic principle of traffic flow dynamics. Then, these features are embedded into the graph nodes and edges and the embedded graph is trained by a message-passing GNN, which yield a communication-based spatially coordinated multi-agent model. Experimental results on some different traveling demand patterns show that the joint action of the proposed model is spatially coordinated so that it improves the network efficiency by rebalancing the traffic demands in the network. Finally, unlike the former cases, there has been a growing interest in developing a data-driven method for the perimeter control. However, few relevant studies can be found in the literature, because there is a lack of delicate macroscopic traffic simulation that can describe the effect of perimeter controls and a lack of basic study on the construction of RL models. Therefore, we introduce RL-approach for the optimization of perimeter control for the first time by applying a newly developed macroscopic traffic simulation based on the network transmission model. As a basic study, we design several perimeter control models using different scope of traffic variables and different methods for parametrizations. Then, we validate the proposed models by evaluating their performances with the test demand scenarios at different levels. The validation results show that the model containing travel demand information adapts to a new demand scenario better than the model containing only density-related factors. This dissertation considers crucial issues for developing an effective signal control model which have not been treated in the previous researches and provides a realistic framework for the traffic signal control. This improves the feasibility of the proposed signal control model and provides a basis for developing a signal control model for a large-scale urban area. Moreover, we suggest a direction for the next-generation signal control systems by leaving an idea for a hierarchical strategy of traffic signal control for a large-scale metropolitan area is left as s future study on the basis of the result of this dissertation.