Feature Level Sensor Fusion for Passive RF and EO Information Integration

Many different sensing modalities across the spectrum exist for collecting and processing data for the purposes of target detection, tracking and differentiation. However, each of these individual modalities from the electromagnetic spectrum contain benefits, limitations, and sources of uncertainty. While research has been conducted to integrate complementary data collected by electro-optical (EO) and radio frequency (RF) modalities, the processing of RF data usually applies traditional methods, such as Doppler. This paper explores the viability of using histogram of I/Q (in-phase and quadrature) data for the purposes of augmenting the detection accuracy that EO input alone is incapable of achieving. Processing the histogram of I/Q data via deep learning, enhances feature resolution for neural network fusion. Using the simulated data from the Digital Imaging and Remote Sensing Image Generation (DIRSIG) dataset, the resulting fusion of EO/RF neural network (FERNN) can achieve 95% accuracy in vehicle detection and scenario categorization, which is a 23% improvement over the accuracy achieved by a standalone EO sensor.