Link between Facial Expressions and Emotional States Induced by Exposure to Multimedia Content

The explosive growth of digital videos has created new challenges for computer science. While many advances on video indexing, retrieval and summarization based on general, subject-independent, objective descriptors have been made in the past years, research on the use of individual subjective preferences and affective states is at the forefront of research and poses great challenges. In this article, we study the relationship between emotional states reported by viewers and their facial physiological changes observed during the display of different video genres. A dataset of twenty videos was created from YouTube video sharing platform. During the exhibition of the videos, the viewer's facial activities have been recorded and analyzed by means of Action Units (AUs). After that, emotional states self-reported by the viewers were assigned to video shots. Labels were divided into four categories, defined according to a discrete version of Russel's Circumplex emotion model. Different machine learning models were trained to test the relationship between the measured facial features and the self-reported emotional categories. We obtained k-fold cross validation accuracies that were above chance for the best learned models. As a result of this study, we concluded that AUs can indeed be used as an valuable tool to estimate emotional categories during exposure to audiovisual stimuli, and, therefore, should be used in further studies that take advantage of those categories to devise personalized multimedia retrieval and summarization approaches.