MMGT: Multimodal Graph-Based Transformer for Pain Detection

Pain can be expressed from multiple modalities, such as facial expressions, physiological signals, and behaviors. For that reason, multimodal learning can greatly benefit automatic pain detection and, more generally, a variety of tasks in the field of affective computing. In this context, as one of our main contributions, we leverage the multimodal interaction among the intermediate modality representations, which are rarely exploited in existing works. In order to capture the relationships between multiple modalities, we propose the Multimodal Graph-based Transformer (MMGT), in which uni-modality feature extraction is performed using Transformers and then fused using a Graph Convolutional Network (GCN). We evaluated MMGT on the BP4D+ dataset, and the results demonstrate the efficiency of our fusion framework for the task of pain detection, which outperformed all the existing approaches under multimodal settings. Our best results were obtained using 2D facial landmarks, action units, and physiological data, on which we achieved 94.95% and 94.91% of accuracy and F1-score, respectively.