Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios
arxiv(2024)
摘要
In the Massive Open Online Courses (MOOC) learning scenario, the semantic
information of instructional videos has a crucial impact on learners' emotional
state. Learners mainly acquire knowledge by watching instructional videos, and
the semantic information in the videos directly affects learners' emotional
states. However, few studies have paid attention to the potential influence of
the semantic information of instructional videos on learners' emotional states.
To deeply explore the impact of video semantic information on learners'
emotions, this paper innovatively proposes a multimodal emotion recognition
method by fusing video semantic information and physiological signals. We
generate video descriptions through a pre-trained large language model (LLM) to
obtain high-level semantic information about instructional videos. Using the
cross-attention mechanism for modal interaction, the semantic information is
fused with the eye movement and PhotoPlethysmoGraphy (PPG) signals to obtain
the features containing the critical information of the three modes. The
accurate recognition of learners' emotional states is realized through the
emotion classifier. The experimental results show that our method has
significantly improved emotion recognition performance, providing a new
perspective and efficient method for emotion recognition research in MOOC
learning scenarios. The method proposed in this paper not only contributes to a
deeper understanding of the impact of instructional videos on learners'
emotional states but also provides a beneficial reference for future research
on emotion recognition in MOOC learning scenarios.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要