F-LMM: Grounding Frozen Large Multimodal Models
CoRR(2024)
摘要
Endowing Large Multimodal Models (LMMs) with visual grounding capability can
significantly enhance AIs' understanding of the visual world and their
interaction with humans. However, existing methods typically fine-tune the
parameters of LMMs to learn additional segmentation tokens and overfit
grounding and segmentation datasets. Such a design would inevitably cause a
catastrophic diminution in the indispensable conversational capability of
general AI assistants. In this paper, we comprehensively evaluate
state-of-the-art grounding LMMs across a suite of multimodal question-answering
benchmarks, observing pronounced performance drops that indicate vanishing
general knowledge comprehension and weakened instruction following ability. To
address this issue, we present F-LMM – grounding frozen off-the-shelf LMMs in
human-AI conversations – a straightforward yet effective design based on the
fact that word-pixel correspondences conducive to visual grounding inherently
exist in the attention weights of well-trained LMMs. Using only a few trainable
CNN layers, we can translate word-pixel attention weights to mask logits, which
a SAM-based mask refiner can further optimise. Our F-LMM neither learns special
segmentation tokens nor utilises high-quality grounded instruction-tuning data,
but achieves competitive performance on referring expression segmentation and
panoptic narrative grounding benchmarks while completely preserving LMMs'
original conversational ability. Additionally, with instruction-following
ability preserved and grounding ability obtained, our F-LMM can perform visual
chain-of-thought reasoning and better resist object hallucinations.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要