ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
arxiv(2024)
摘要
The integration of Multimodal Large Language Models (MLLMs) with robotic
systems has significantly enhanced the ability of robots to interpret and act
upon natural language instructions. Despite these advancements, conventional
MLLMs are typically trained on generic image-text pairs, lacking essential
robotics knowledge such as affordances and physical knowledge, which hampers
their efficacy in manipulation tasks. To bridge this gap, we introduce
ManipVQA, a novel framework designed to endow MLLMs with Manipulation-centric
knowledge through a Visual Question-Answering format. This approach not only
encompasses tool detection and affordance recognition but also extends to a
comprehensive understanding of physical concepts. Our approach starts with
collecting a varied set of images displaying interactive objects, which
presents a broad range of challenges in tool object detection, affordance, and
physical concept predictions. To seamlessly integrate this robotic-specific
knowledge with the inherent vision-reasoning capabilities of MLLMs, we adopt a
unified VQA format and devise a fine-tuning strategy that preserves the
original vision-reasoning abilities while incorporating the new robotic
insights. Empirical evaluations conducted in robotic simulators and across
various vision task benchmarks demonstrate the robust performance of ManipVQA.
Code and dataset will be made publicly available at
https://github.com/SiyuanHuang95/ManipVQA.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要