Radio Resource Management for Cellular-Connected UAV: A DRL Solution

Integrating unmanned aerial vehicles (UAVs) into the existing cellular networks faces lots of challenges, in which one of the most striking concerns is how to adopt UAVs into cellular networks with less adverse effects on ground user equipments (UEs). In this paper, a cellular-connected UAV network is considered, where multiple UAVs receive messages from terrestrial BSs in the down-link, while BSs are serving ground users in their cells. To enhance wireless transmission quality for UAVs while protecting ground UEs from being interfered, a joint time-frequency resource block (RB) and beamforming optimization problem minimizing the ergodic outage duration (EOD) of UAV is investigated. To solve the proposed radio resource management problem, a deep reinforcement learning (DRL) solution is proposed, where deep double duelling Q network (D3QN) and twin delayed deep deterministic policy gradient (TD3) are invoked to deal with RB allocation in discrete action domain and beamforming design in continuous action regime, respectively. The hybrid D3QN-TD3 solution is applied to solve the outer MDP and the inner MDP interactively so that it can achieve the sub-optimal result for the considered optimization problem. Simulation results have illustrated the effectiveness of the proposed hybrid D3QN-TD3 algorithm, compared to exhaustive/random search based benchmarks.