Modeling Emotional Cue Integration of Context-rich and Dynamic Stimuli reveals Bayesian as well as anti-Bayesian Properties

Humans often encounter multiple complex social cues when perceiving emotion, but how does the brain combine cues into a single judgment of emotion? Observers do not just use facial expressions, as previous studies have found that contextual information is just as important (Chen and Whitney, PNAS, 2019). One idea suggests that emotional cues are combined using a Bayesian framework where cues are weighted by their reliability (Zaki 2013, Ong et al. 2015). In our study, we investigated emotional cue integration using context-rich and dynamic stimuli, which mimic the natural properties of images present when humans infer emotion in their everyday lives. In the experiment, observers continuously tracked and inferred the affect of a target character while watching a video. Observers were organized into three conditions: (1) the context-only condition which had the target character masked-out, (2) the character-only condition which had the context masked-out, and (3) the ground-truth condition where no visual information was masked-out. We used a Bayesian model to integrate the group-averaged context-only and character-only ratings of valence and arousal to model the empirical group-averaged ground-truth ratings. Using bootstrapping and 95% confidence intervals, we found that the Bayesian model greatly outperformed observers' character-only and context-only ratings in matching the empirical group-averaged ground-truth ratings. Comparing our model to a linear integration model revealed that the Bayesian model performed better for some videos while the linear model performed better on others. Individual video analysis revealed that around half of the ratings involved “anti-Bayesian” combinations of the context and character ratings (an amplification of ratings, a reduction of ratings, and sign flipping of ratings rather than merging). Our results suggest that observers combine multiple cues when favorable, but this combination does not consistently follow a Bayesian framework. Instead, there are conditions in which “anti-Bayesian” approaches for emotion perception may be required.