Optimization of Energy Efficiency for Uplink mURLLC Over Multiple Cells Using Cooperative Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (RL) has recently been adopted to solve massive ultra-reliable and low-latency communications (mURLLC) energy efficiency (EE) optimization problem in a single-cell cellular network under random access. Bursty traffic is an important characteristic of mURLLC users (UEs). This characteristic and its impact on the RL scheme are generally ignored in many RL-based studies related to the optimization of EE for uplink mURLLC. Moreover, in a smart factory with multiple cells, inter-cell interference and shadow fading further complicate EE optimization. To address these issues, we propose a novel cooperative multi-agent scheme to maximize the long-term EE in a multi-cell cellular network with mURLLC bursty traffic and a K-repetition scheme by optimizing the repetition value and transmission power. A UE clustering algorithm and an intermittent learning mode are adopted to reduce the computational complexity and mitigate the impact of bursty traffic on the RL scheme. A proper reward function is designed to address both long-term EE maximization and the number of successfully served UEs under high reliability requirement. The simulation results show that our proposed cooperative multi-agent reinforcement learning scheme greatly outperforms other existing schemes in terms of long-term accumulated EE and the number of successfully served UEs.