Multimodal Multilevel Fusion for Sequential Protective Behavior Detection and Pain Estimation

In this paper, we present our approach to the FG 2020 EmoPain Challenge for tasks 2 (pain estimation) and 3 (protective behavior detection) from multimodal movement data. We propose to perform sequential protective behavior detection and pain estimation using human movement information. First, we predict the existence of pain, and then use this information along with the multimodal movement data for protective behavior detection. Finally, this information is fused to estimate level of pain. In this work, we apply both early fusion (feature fusion including metadata, modalities, exercises and probabilities) and post-fusion (decision fusion). The proposed approach is encouraging, as it outperforms the baseline, with high margin for both pain estimation and protective behavior detection on the EmoPain challenge 2020 dataset.