Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning
arxiv(2023)
摘要
Designing an effective representation learning method for multimodal
sentiment analysis tasks is a crucial research direction. The challenge lies in
learning both shared and private information in a complete modal
representation, which is difficult with uniform multimodal labels and a raw
feature fusion approach. In this work, we propose a deep modal shared
information learning module based on the covariance matrix to capture the
shared information between modalities. Additionally, we use a label generation
module based on a self-supervised learning strategy to capture the private
information of the modalities. Our module is plug-and-play in multimodal tasks,
and by changing the parameterization, it can adjust the information exchange
relationship between the modes and learn the private or shared information
between the specified modes. We also employ a multi-task learning strategy to
help the model focus its attention on the modal differentiation training data.
We provide a detailed formulation derivation and feasibility proof for the
design of the deep modal shared information learning module. We conduct
extensive experiments on three common multimodal sentiment analysis baseline
datasets, and the experimental results validate the reliability of our model.
Furthermore, we explore more combinatorial techniques for the use of the
module. Our approach outperforms current state-of-the-art methods on most of
the metrics of the three public datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要