A Design of Spectrally-Efficient Low-Complexity QAM-FBMC Systems With Mismatched Prototype Filters

In this article, we propose a low-complexity quadrature amplitude modulation (QAM) filter-bank multicarrier (FBMC) system with high spectral efficiency and low self interference. To circumvent the limitations imposed by the BalianLow theorem on the interference suppression capability of QAMFBMC systems, the proposed system relaxes time-frequency (TF) product at the transmitter (TX) and time-domain localization at the receiver (RX). The PHYDYAS filter is employed as a TX prototype filter, and a filter mismatched to the TX filter is employed as an RX prototype filter. Since the PHYDYAS filter has excellent spectral confinement, its use well compensates for the loss in TF-product and leads to higher spectral efficiency than the conventional systems in multi-user uplink. The RX has an extended observation window and a separate equalizer followed by a fixed pre-computed filter. To find the RX prototype filter that minimizes the self-interference, an optimization problem is formulated based on a bi-orthogonality condition, subject to the frequency-domain sparsity constraint in terms of the l(0)-norm. To efficiently search for the solution, a modified relaxed LASSO procedure is proposed, where the size of the problem is reduced to make the problem numerically tractable. Numerical results and discussions show that the proposed QAMFBMC system strikes an excellent balance between performance and complexity.