SAR Self-Enhanced by Electro-optical Network (SARSEEN)

In this work, we investigate the relationship between paired synthetic aperture radar (SAR) and optical images. SAR sensors have the capabilities of penetrating clouds and capturing data at night, whereas optical sensors cannot. Due to this, we are particularly interested in the case where we have access to both modalities during training, but only the SAR during test time. Our goal is to improve the performance of SAR only models by learning a self-generated prior about optical imagery. To that end, we developed a framework that consists of a deep neural network that inputs a SAR image and predicts a Canny edge map of the optical image. The Canny edge map retains structural information about the optical image, while removing superfluous high frequency details that is hard to generate from the SAR image. For downstream tasks, we concatenate the predicted edge map to the original SAR image and use that as the input. Our experiments show that by using this predicted edge map as additional information, we can outperform the same model that is only given the SAR image.