What is the Visual Cognition Gap between Humans and Multimodal LLMs?
CoRR(2024)
摘要
Recently, Multimodal Large Language Models (MLLMs) have shown great promise
in language-guided perceptual tasks such as recognition, segmentation, and
object detection. However, their effectiveness in addressing visual cognition
problems that require high-level reasoning is not well-established. One such
challenge is abstract visual reasoning (AVR) – the cognitive ability to
discern relationships among patterns in a set of images and extrapolate to
predict subsequent patterns. This skill is crucial during the early
neurodevelopmental stages of children. Inspired by the AVR tasks in Raven's
Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC),
we propose a new dataset MaRs-VQA and a new benchmark VCog-Bench containing
three datasets to evaluate the zero-shot AVR capability of MLLMs and compare
their performance with existing human intelligent investigation. Our
comparative experiments with different open-source and closed-source MLLMs on
the VCog-Bench revealed a gap between MLLMs and human intelligence,
highlighting the visual cognitive limitations of current MLLMs. We believe that
the public release of VCog-Bench, consisting of MaRs-VQA, and the inference
pipeline will drive progress toward the next generation of MLLMs with
human-like visual cognition abilities.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要