Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
arxiv(2024)
摘要
In vision-language pre-training (VLP), masked image modeling (MIM) has
recently been introduced for fine-grained cross-modal alignment. However, in
most existing methods, the reconstruction targets for MIM lack high-level
semantics, and text is not sufficiently involved in masked modeling. These two
drawbacks limit the effect of MIM in facilitating cross-modal semantic
alignment. In this work, we propose a semantics-enhanced cross-modal MIM
framework (SemMIM) for vision-language representation learning. Specifically,
to provide more semantically meaningful supervision for MIM, we propose a local
semantics enhancing approach, which harvest high-level semantics from global
image features via self-supervised agreement learning and transfer them to
local patch encodings by sharing the encoding space. Moreover, to achieve deep
involvement of text during the entire MIM process, we propose a text-guided
masking strategy and devise an efficient way of injecting textual information
in both masked modeling and reconstruction target acquisition. Experimental
results validate that our method improves the effectiveness of the MIM task in
facilitating cross-modal semantic alignment. Compared to previous VLP models
with similar model size and data scale, our SemMIM model achieves
state-of-the-art or competitive performance on multiple downstream
vision-language tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要