ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models
CoRR(2024)
摘要
Recent advancements in AI have led to the development of large multimodal
models (LMMs) capable of processing complex tasks involving joint reasoning
over text and visual content in the image (e.g., navigating maps in public
places). This paper introduces ConTextual, a novel benchmark comprising
instructions designed explicitly to evaluate LMMs' ability to perform
context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
real-world scenarios (e.g., time-reading, navigation, shopping and more)
demanding a deeper understanding of the interactions between textual and visual
elements. Our findings reveal a significant performance gap of 30.8
the best-performing LMM, GPT-4V(ision), and human capabilities using human
evaluation indicating substantial room for improvement in context-sensitive
text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
categories like meme and quote interpretation, its overall performance still
lagged behind humans. In addition to human evaluations, we also employed
automatic evaluation metrics using GPT-4, uncovering similar trends in
performance disparities. We also perform a fine-grained evaluation across
diverse visual contexts and provide qualitative analysis which provides a
robust framework for future advancements in the LMM design.
https://con-textual.github.io/
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要