DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever
CoRR(2024)
摘要
Recently, substantial advancements in pre-trained vision-language models have
greatly enhanced the capabilities of multi-modal dialog systems. These models
have demonstrated significant improvements by fine-tuning on downstream tasks.
However, the existing pre-trained models primarily focus on effectively
capturing the alignment between vision and language modalities, often ignoring
the intricate nature of dialog context. In this paper, we propose a
parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog
retrieval. Specifically, our approach introduces a multi-modal context prompt
generator to learn context features which are subsequently distilled into
prompts within the pre-trained vision-language model CLIP. Besides, we
introduce domain prompt to mitigate the disc repancy from the downstream dialog
data. To facilitate various types of retrieval, we also design multiple experts
to learn mappings from CLIP outputs to multi-modal representation space, with
each expert being responsible to one specific retrieval type. Extensive
experiments show that DialCLIP achieves state-of-the-art performance on two
widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a
mere 0.04
efficiency of our proposed approach, underscoring its potential to advance the
field of multi-modal dialog retrieval.
更多查看译文
关键词
multi-modal dialogue,response selection
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要