Reconstructing the neurodynamics of face perception during real world vision in humans using intracranial EEG recordings

We use face perception to see and understand people around us during natural behavior in the real world. Here, we take advantage of the unique opportunity afforded by intracranial recordings in two epilepsy patients to assess the neural basis of face perception during natural, unscripted interactions in real world settings with friends, family, and experimenters. With eye tracking glasses, we captured what subjects saw, time locked to the corresponding neural activity, on a fixation-by-fixation basis for hours during these interactions. We restricted the analysis to face fixations annotated using a combination of manual annotations and computer vision. After training a bidirectional Canonical Component Analysis (CCA) model on training fixations, we sought to reconstruct an image of the face people were seeing based on the corresponding pattern of neural activity, and reconstruct an image of the neural activity based on the corresponding face image, on a fixation-by-fixation basis in a left out test sample of fixations. Significant reconstruction of both the face image subjects were seeing (out of sample R= 0.46; 0.26) and neural activity (out of sample R= 0.29; 0.14) was observed. By assessing which features are reconstructed accurately, we find that parietal, temporal and occipital cortices around 200 ms after fixation onset are important for face processing during natural social interactions. Individual Canonical Components of the model enable a more granular breakdown to examine which specific face features are coded by which particular aspects of neural activity. We will use this approach to test norm based and metric code models of face perception during natural face perception. Our results lay the foundation for understanding the neural basis of visual perception during natural behavior in the real world.