TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection
CoRR(2024)
Abstract
Task-oriented object detection aims to find objects suitable for
accomplishing specific tasks. As a challenging task, it requires simultaneous
visual data processing and reasoning under ambiguous semantics. Recent
solutions are mainly all-in-one models. However, the object detection backbones
are pre-trained without text supervision. Thus, to incorporate task
requirements, their intricate models undergo extensive learning on a highly
imbalanced and scarce dataset, resulting in capped performance, laborious
training, and poor generalizability. In contrast, we propose TaskCLIP, a more
natural two-stage design composed of general object detection and task-guided
object selection. Particularly for the latter, we resort to the recently
successful large Vision-Language Models (VLMs) as our backbone, which provides
rich semantic knowledge and a uniform embedding space for images and texts.
Nevertheless, the naive application of VLMs leads to sub-optimal quality, due
to the misalignment between embeddings of object images and their visual
attributes, which are mainly adjective phrases. To this end, we design a
transformer-based aligner after the pre-trained VLMs to re-calibrate both
embeddings. Finally, we employ a trainable score function to post-process the
VLM matching results for object selection. Experimental results demonstrate
that our TaskCLIP outperforms the state-of-the-art DETR-based model TOIST by
3.5
inference.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined