Multi-modal Affect Detection Using Thermal Imaging in a Gamified Exercise

Abstract Affect recognition, or the ability to detect and interpret emotional states, has the potential to be a valuable tool in the field of healthcare. In particular, it can be useful in gamified therapy, which involves using gaming techniques to motivate and keep the engagement of patients in therapeutic activities. This study aims to examine the effectiveness of a tangible PacMan game in inducing different emotions in participants and the accuracy of machine learning models using thermal imaging and action unit data for affect classification. A self-report survey and three machine learning models were used to assess emotions including frustration, boredom, and enjoyment in participants during different phases of the game. The results showed that the game was successful in inducing significant differences in enjoyment and frustration, but only enjoyment and frustration were significantly different from boredom. The machine learning models showed that the multimodal approach with the combination of thermal imaging and action units had the highest accuracy of 77% for emotion classification in the 7-second window, while thermal imaging had the lowest standard deviation among participants. The results suggest that thermal imaging and action units can be effective in detecting affective states and might have the potential to be used in healthcare applications, such as gamified therapy, as a promising non-intrusive method for detecting internal states.