Pheromone-Enhanced Reinforcement Learning for Multi-Agent Self Organization in Partially Observable Non-Stationary Environments

Abstract

A complex system refers to a collection of interconnected components or agents that interact with each other in nonlinear ways. Reinforcement learning (RL) is an optimization technique where an agent learns to make decisions through feedback signals obtained by interacting with its environment. It is well-suited for real-world problems as it can learn without needing a model of the environment. Although, as the number of agents in a system grows, centralized control with RL becomes impractical due to scalability limitations.

This scalability issue prompts the need for decentralized control with multi-agent reinforcement learning (MARL). However, MARL faces a deep-rooted challenge in coordinating actions of multiple agents to achieve common objectives in partially observable non-stationary environments. Some complex multi-agent systems (MAS) in nature like ant colonies, bee swarms or bird flocks can self-organize and exhibit emergent coordinated behaviour. For example, during foraging of food, ants can emit chemical pheromones that serve as an indirect form of communication (known as stigmergy) and influence the movement of other ants perceiving this chemical, displaying collective swarm behaviour.

To address the challenge of coordination in MARL for partially observable and non-stationary environments, this thesis proposes Pheromone-Enhanced Reinforcement Learning (PERL), an approach that adopts stigmergic communication mechanisms from Swarm Intelligence (SI) methods like Ant Colony Optimization (ACO). PERL integrates digital pheromones and stigmergy with MARL. Stigmergic mechanisms, like evaporation, ensure the assimilation of new information, while diffusion facilitates the transmission of pheromones throughout the entire system, promoting self-organization at a systemic level. This results in emergent behavior arising from individual agent interactions, enabling coordination amidst partially observable and non-stationary conditions.

To tackle the challenges of a real-world complex system characterized by partial observability and non-stationarity, demanding effective coordination, we conducted evaluations, we evaluated PERL in managing traffic with Connected and Autonomous Vehicles (CAVs). Our assessment covered a range of traffic scenarios, including multi-CAV navigation involving lane changes amidst dynamic obstacles on multilane highways, as well as the negotiation of single and interconnected four-way intersections with conflicting unidirectional unilane flows. The diversity of tested applications demonstrates that PERL is application-agnostic.

Our baselines include two distinct rule-based or heuristic approaches, and MARL without the use of communication strategies (referred to as Independent RL or IRL). Empirical studies demonstrates that PERL outperforms all baseline approaches. The time taken for all agents to complete the simulation and mean waiting time is better than baselines by atleast ~13% and ~24%, respectively, in the intersection scenario. Additionally, PERL's mean recovery time after perturbations is also better by atleast ~18% in highway lane-changing scenarios with dynamic blockages.

The feedback signal proposed by PERL incentivizes minimizing deviation in local pheromone levels. Empirical results show that this translates to an objective to achieve a system-wide equitable distribution of pheromones, through stigmergic mechanisms integrated into PERL, and emerges as a self-organizing behavior that aims to redistribute traffic, which is seen to improve overall traffic flow. Furthermore, the evaluation results demonstrate that PERL exhibits a tendency to approach global pheromone equilibrium ~20% earlier than its baseline counterparts.