Generalized Eigenvalue Based Detection of Signals in Colored Noise: A Sample Deficient Analysis

This paper investigates the signal detection problem in colored noise with an unknown covariance matrix. To be specific, we consider a scenario in which the number of signal bearing samples (n) is strictly smaller than the dimensionality of the signal space (m). Our test statistic is the leading generalized eigenvalue of the whitened sample covariance matrix (a.k.a. F-matrix) which is constructed by whitening the signal bearing sample covariance matrix with noise-only sample covariance matrix. The sample deficiency (i.e., m > n) in turn makes this F-matrix rank deficient, thereby singular. Therefore, an exact statistical characterization of the leading generalized eigenvalue (l.g.e.) of a singular F-matrix is of paramount importance to assess the performance of the detector (i.e., the receiver operating characteristics (ROC)). To this end, we employ the powerful orthogonal polynomial approach to derive a new finite dimensional c.d.f. expression for the l.g.e. of a singular F-matrix. It turns out that when the noise only sample covariance matrix is nearly rank deficient and the signal-to-noise ratio is O(m), the ROC profile converges to a limit.