Deep Learning-Based Detection Algorithm for the Multi-User MIMO-NOMA System

Recently, non-orthogonal multiple access (NOMA) has become prevalent in 5G communication. However, the traditional successive interference cancellation (SIC) receivers for NOMA still encounter challenges. The near-far effect between the users and the base stations (BS) results in a higher bit error rate (BER) for the SIC receiver. Additionally, the linear detection algorithm used in each SIC stage fails to eliminate the interference and is susceptible to error propagation. Consequently, designing a high-performance NOMA system receiver is a crucial challenge in NOMA research and particularly in signal detection. Focusing on the signal detection of the receiver in the NOMA system, the main work is as follows. (1) This thesis leverages the strengths of deep neural networks (DNNs) for nonlinear detection and incorporates the low computational complexity of the successive interference cancellation (SIC) structure. The proposed solution introduces a feedback deep neural network (FDNN) receiver to replace the SIC in signal detection. By employing a deep neural network for nonlinear detection at each stage, the receiver mitigates error propagation, lowers the BER in NOMA systems, and enhances resistance against inter-user interference (IUI). (2) We describe its algorithm flow and provide simulation results comparing FDNN and SIC receivers under MIMO-NOMA scenarios. The simulations clearly demonstrate that FDNN receivers outperform SIC receivers in terms of BER for MIMO-NOMA systems.