Synthetic sparse nested array with extended aperture and reduced mutual coupling for DOA estimation

•A dual-polarization synthetic nested array composed of three single-polarization subarrays is proposed with limited physical antennas.•For the proposed array, the inter-element spacings of subarrays are extended by D-fold, to reduce the mutual coupling between adjacent antennas.•In the proposed array, most differently polarized antennas are located at different positions to reduce the mutual coupling between them.•Based on the coarse estimation and fine but ambiguous estimation, an automatic pair matching is proposed to obtain unique angle estimation.