Effects of Demographics and Photometric Normalization on Image Translation GANs for Cross-Spectral Face Recognition

This paper focuses on thermal-to-visible face matching through image synthesis. Most of the legacy face image datasets are composed of visible band data. Thermal band as well as dual band, i.e. visible and thermal face datasets, are limited. Operating in the thermal band and therefore working on visible thermal face recognition (FR) systems can be beneficial in various scenarios. The challenge is cross-spectral matching, i.e. matching gallery, visible band, face images against thermal ones. To address this problem, we train and test two of the most popular image-to-image translation Generative Adversarial Networks (GANs). These are Pix2pix and StarGAN2. In this work, the two aforementioned GAN trained models are tested, and the visible face images generated are matched against the ground truth visible faces using one of the most powerful visible-to-visible face matching algorithms, namely Facenet. We also perform an ablation study where the original thermal and visible images are photometrically normalized before training the image synthesis-specific models. The main outcomes of our study are that the FR accuracy from the pix2pix model did not vary significantly; when using StarGAN2, the original face images yield much higher accuracy compared to the photometrically normalized ones; finally, we observe that, when using the pix2pix model for image synthesis, bearded and non-Caucasian generated face images suffer the most from different noise factors. Specifically, the FR accuracy when using pix2pix after image synthesis yields a face verification area under curve (AUC) of 58.3%, while the same models when tested on data excluding bearded and non-Caucasian faces yields an accuracy of 68.6%.