HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision
CoRR(2024)
Abstract
Large Vision Language Models (VLMs) are now the de facto state-of-the-art for
a number of tasks including visual question answering, recognising objects, and
spatial referral. In this work, we propose the HOI-Ref task for egocentric
images that aims to understand interactions between hands and objects using
VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9M
question-answer pairs for training and evaluating VLMs. HOI-QA includes
questions relating to locating hands, objects, and critically their
interactions (e.g. referring to the object being manipulated by the hand). We
train the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Our
results demonstrate that VLMs trained for referral on third person images fail
to recognise and refer hands and objects in egocentric images. When fine-tuned
on our egocentric HOI-QA dataset, performance improves by 27.9
hands and objects, and by 26.7
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined