Hierarchical Multi-agent Deep Reinforcement Learning for System-wide Urban Traffic Optimization

Abstract

Rapid urbanization has led to a substantial rise in traffic volumes and the expansion of road networks, creating large and complex urban road transportation systems. Existing traffic-light systems, however, lack the adaptability to effectively manage dynamic traffic conditions. Variations in intersection geometry, lane configurations, and traffic flow can overwhelm these systems, resulting in congestion, lower throughput, unpredictable journey times, and increased emissions and fuel consumption. Consequently, an adaptive and generalized traffic-light-free approach to urban traffic management holds significant potential to improve traffic flow by exploiting the deployment of connected and automated vehicles (CAVs) in large-scale traffic networks.

Effectively managing urban traffic demands a coordinated and adaptive approach, considering the presence of numerous interconnected intersections with distinct geometric layouts and varying traffic volumes. This diversity makes developing a generalized and scalable solution particularly challenging. The large scale of urban road networks requires solutions that ensure not only efficiency at individual intersections but also optimize system-wide performance. Additionally, adapting to real-time traffic variations while maintaining computational efficiency places significant demands on potential solutions.

From a research perspective, urban-scale traffic optimization remains dominated by variants of traffic light control methods, with unsignalized intersection optimization at such scales largely unexplored. Recent studies on unsignalized intersection management are broadly categorized into heuristic-based and learning-based methods, but both face challenges in managing large-scale and heterogeneous networks. Heuristic-based policies struggle with high traffic demand and unforeseen variations, requiring extensive computational resources and being constrained by platoon size limitations. Learning-based approaches leveraging CAVs show promise on isolated intersections but remain insufficiently explored for large-scale scenarios, focusing primarily on basic collision avoidance. This highlights the need for new scalable strategies to manage networks of unsignalized intersections effectively.

This thesis addresses this gap by introducing a hierarchical multi-agent deep reinforcement learning (MADRL) optimization framework for large-scale, unsignalized, and heterogeneous urban road networks. Leveraging the coordination advantages of CAVs, the solution employs a three-tiered control structure to optimize traffic flow. At the lowest level, precise trajectories are calculated for CAVs. The middle layer employs parallel neural Monte-Carlo tree search (PNMCTS) to evaluate intersection crossing schedules simultaneously, selecting the optimal schedule for collision-free crossing. At the highest level, a decentralized MADRL system coordinates multiple intersections across the network, grouping vehicles into platoons of varying lengths to maximize throughput. This integrated approach aims to provide a scalable, efficient and safe solution capable of optimizing traffic flow across heterogeneous urban networks under diverse traffic conditions.

We evaluate the proposed system through a comprehensive set of experiments that assess each level of the hierarchical framework. The evaluation encompasses both homogeneous and heterogeneous intersection networks, along with large-scale simulations of Dublin city's road network. Using zero-shot and few-shot learning, we examine the system's generality and scalability across diverse urban environments.

The experimental results highlight that the proposed hierarchical MADRL framework significantly outperforms traffic-light-based systems, providing notable improvements in average travel times, travel-time reliability, and intersection throughput across diverse urban traffic scenarios. These findings underscore the potential of the hierarchical, multi-agent strategy to reduce congestion, improve intersection performance in large-scale, non-stationary urban environments. The proposed approach advances traffic management by integrating precise trajectory control, PNMCTS cross scheduling, and MADRL for scalable, adaptive, and system-wide optimization, providing a comprehensive solution to the dynamic challenges of future urban traffic systems.