Text-Based Reasoning About Vector Graphics
CoRR(2024)
Abstract
While large multimodal models excel in broad vision-language benchmarks, they
often struggle with tasks requiring precise perception of low-level visual
details, such as comparing line lengths or solving simple mazes. In particular,
this failure mode persists in question-answering tasks about vector graphics –
images composed purely of 2D objects and shapes. To address this challenge, we
propose the Visually Descriptive Language Model (VDLM), which performs
text-based reasoning about vector graphics. VDLM leverages Scalable Vector
Graphics (SVG) for a more precise visual description and first uses an
off-the-shelf raster-to-SVG algorithm for encoding. Since existing language
models cannot understand raw SVGs in a zero-shot setting, VDLM then bridges SVG
with pretrained language models through a newly introduced intermediate
symbolic representation, Primal Visual Description (PVD), comprising primitive
attributes (e.g., shape, position, measurement) with their corresponding
predicted values. PVD is task-agnostic and represents visual primitives that
are universal across all vector graphics. It can be learned with procedurally
generated (SVG, PVD) pairs and also enables the direct use of LLMs for
generalization to complex reasoning tasks. By casting an image to a text-based
representation, we can leverage the power of language models to learn alignment
from SVG to visual primitives and generalize to unseen question-answering
tasks. Empirical results show that VDLM achieves stronger zero-shot performance
compared to state-of-the-art LMMs, such as GPT-4V, in various low-level
multimodal perception and reasoning tasks on vector graphics. We additionally
present extensive analyses on VDLM's performance, demonstrating that our
framework offers better interpretability due to its disentangled perception and
reasoning processes. Project page: https://mikewangwzhl.github.io/VDLM/
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined