PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter
CoRR(2024)
摘要
This paper demonstrates that a progressively aligned language model can
effectively bridge frozen vision encoders and large language models (LLMs).
While the fundamental architecture and pre-training methods of vision encoders
and LLMs have been extensively studied, the architecture and training strategy
of vision-language adapters vary significantly across recent works. Our
research undertakes a thorough exploration of the state-of-the-art perceiver
resampler architecture and builds a strong baseline. However, we observe that
the vision-language alignment with perceiver resampler exhibits slow
convergence and limited scalability with a lack of direct supervision. To
address this issue, we propose PaLM2-VAdapter, employing a progressively
aligned language model as the vision-language adapter. Compared to the strong
baseline with perceiver resampler, our method empirically shows faster
convergence, higher performance, and stronger scalability. Extensive
experiments across various Visual Question Answering (VQA) and captioning tasks
on both images and videos demonstrate that our model exhibits state-of-the-art
visual understanding and multi-modal reasoning capabilities. Notably, our
method achieves these advancements with 30 70
state-of-the-art large vision-language models, marking a significant efficiency
improvement.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要