Beyond superficial emotion recognition: Modality-adaptive emotion recognition system

With the rapid development of deep learning, emotion recognition from facial expressions has also greatly improved, but there are still limitations in terms of reliability when applied to the real world. In other words, facial expressions and the corresponding true emotions may be inconsistent. So, although audio or bio signals are additionally used to improve the reliability of emotion recognition, reasonable estimation performance or real-time operation of emotion recognition is not guaranteed yet. This paper addresses both of the above challenging issues. Each sensor input and feature extraction process is totally and asynchronously processed through a parallel processing library. And then, we predict the comprehensive state of the subject's internal/external emotions through modality-adaptive fusion, which considers the influence of the features of each modality. We verified the performance of the proposed system through a real-time pilot test, and achieved accuracy of up to 33% higher than emotion recognition using only external signals, i.e., video and audio.