Optimising Energy Efficiency in UAV-Assisted Networks using Multi-Agent Reinforcement Learning - Omoniwa, Babatunji (Trinity College Dublin. School of Computer Science & Statistics. Discipline of Computer Science, 2023)
The demand for cellular connectivity continues to witness unprecedented growth over the years. Unmanned Aerial Vehicles (UAVs) equipped with small cells can provide ubiquitous connectivity to static and mobile ground users in situations of increased network demand or points of failure in existing terrestrial cellular infrastructure. We consider a system called a UAV-assisted network that uses UAVs to serve ground users. However, UAVs deplete energy while hovering in the sky and providing coverage for extended periods of time. Furthermore, multiple UAVs sharing the same frequency spectrum and deployed to provide wireless connectivity to users in a given area may experience a decrease in the system’s energy efficiency (EE) due to interference from neighbouring UAV cells or other access points. Recent approaches focus on optimising the system’s EE by optimising the trajectory of UAVs serving only static ground users and neglecting mobile users. Several others neglect the impact of interference from nearby UAV cells, assuming an interference-free network environment. Furthermore, some works assume global spatial knowledge of ground users’ location via a central controller (CC) that periodically scans the network perimeter and provides real-time updates to the UAVs for decision-making. However, this assumption may be unsuitable in disaster scenarios since it requires significant information exchange between the UAVs and CC. Moreover, it may not be possible to track users’ locations in a disaster scenario. Despite growing research interest in decentralised control over centralised UAVs’ control, collaboration among UAVs to improve the systems’ EE has not been adequately explored. In dynamic environments with changing users’ distribution, it is challenging to track users in real-time without apriori knowledge of the users’ distribution or gaining such insight from a CC. This thesis’ main contribution, the Decentralised Multi-Agent Reinforcement Learning (DMARL), allows each UAV equipped with an autonomous agent to intelligently serve ground users while improving the overall system’s EE. The DMARL attempts to improve the total system’s energy efficiency while providing wireless connectivity to ground users in an interference-limited network environment. Thus, we address this by decomposing the DMARL into five variants. The first variant investigates how multiple UAVs, each with an independent learning agent learn a policy that improves the total system’s energy efficiency while serving static and mobile ground users without the knowledge of the users’ locations from a CC. An agent-controlled UAV can have a wider view of its environment by gaining more knowledge for better decisions when information is exchanged with closest neighbours. Therefore, we propose two modes of collaboration, an indirect and a direct variant (variants 2 and 3, respectively), to improve the system’s EE in a shared, dynamic and interference-limited network environment. The direct collaboration allows UAVs to share their data via existing 3GPP guidelines, while the indirect variant has no such mechanism but implicitly reflects this knowledge in its reward formulation as an incentive towards collaborative behaviours. More importantly, the past coverage performance of UAVs may influence their decision to collaborate while serving users in dense and uneven users’ distribution. Lastly, we propose direct and indirect collaborative variants that allow UAVs to be density-aware by collaborating to intelligently serve densely distributed users (variants 4 and 5, respectively). We perform evaluations under different network configurations. Results show that our DMARL outperforms centralised baselines that assume prior global knowledge of ground users’ location in terms of EE by as much as 80%. When compared to our closest decentralised MARL baseline which neglects the impact of interference when serving pedestrians, we discover that collaboration provides improved systems’ EE by as much as 55% - 75%. In city traffic, motorways and national roads, the DMARL outperforms state-of-the-art MARL approaches which do not account for varying users’ densities in terms of EE by as much as 65% - 98%. These findings demonstrate the effectiveness of our approach in providing UAVs deployed in an environment with the intelligence to provide coverage in an energy-efficient manner.
APPROVED
Keywords: Machine learning; Multi-agent reinforcement learning; UAV-assisted networks; Energy management of UAVs; Decentralised decision making; Wireless connectivity