Enhancing security for physical layer communication in RIS-aided MIMO-NOMA systems in the presence of an eavesdropper

This paper investigates the potential of leveraging a reconfigurable intelligent surface (RIS) in the multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) wireless communication systems. In fact, NOMA improves spectral efficiency (SE) through power domain utilization, and RIS enhances the system performance by manipulating the propagation environment. In this paper, we investigate the physical layer security (PLS) of a RIS-aided MIMO-NOMA system serving two users in the presence of an eavesdropper (EV), where the channel state information (CSI) of the EV is unknown for the BS. To address the security challenge, we utilize secrecy outage probability (SOP) and minimize it by optimizing the RIS’s phase shift matrix, the beamforming vector, and the power allocation coefficients. Due to the non-convexity of the main optimization problem, we decompose the initial problem into three subproblems, and then, we propose an iterative approach to determine the unknown parameters alternately. The first subproblem involves obtaining the optimal beamforming vector and phase shift matrix using the semidefinite relaxation (SDR) method, which introduces high complexity. As an alternative, another algorithm is proposed based on the duality of the optimization problem, offering lower complexity and better accuracy for high-dimensional optimization problems. In this algorithm, the successive convex approximation (SCA) technique is employed to achieve the optimum phase shift matrix, and the stochastic gradient (SG) approach is used to determine the beamforming vector. Furthermore, the paper establishes the optimal values of power allocation coefficients by minimizing the difference between the SOPs of the two users. Finally, the simulation results demonstrate that the proposed power allocation coefficients scheme significantly enhances system performance compared to the existing fixed power allocation coefficients scheme.