ASER analysis of generalized hexagonal QAM schemes for NOMA systems over Nakagami‐ m fading channels

Summary To achieve high data rates expected from beyond 5G communications, higher‐order modulation techniques have been explored. The energy‐efficient modulation technique with a high data rate has encouraged researches towards an optimum two‐dimensional hexagonal‐shaped constellation, namely, hexagonal quadrature amplitude modulation (HQAM). Thus, in this work, we analyze the average symbol error rate (ASER) of HQAM schemes by considering a two‐user nonorthogonal multiple access (NOMA) pair. Closed‐form expressions for ASER of HQAM schemes for users are derived over generalized Nakagami‐ fading channels. Further, for the HQAM constellation feasibility in two‐user downlink NOMA systems, the power allocation criterion for the users is presented. Furthermore, the impact of modulation order of the users over the systems ASER analysis is investigated and valuable insights are drawn.