How is Visual Attention Influenced by Text Guidance? Database and Model
arxiv(2024)
摘要
The analysis and prediction of visual attention have long been crucial tasks
in the fields of computer vision and image processing. In practical
applications, images are generally accompanied by various text descriptions,
however, few studies have explored the influence of text descriptions on visual
attention, let alone developed visual saliency prediction models considering
text guidance. In this paper, we conduct a comprehensive study on text-guided
image saliency (TIS) from both subjective and objective perspectives.
Specifically, we construct a TIS database named SJTU-TIS, which includes 1200
text-image pairs and the corresponding collected eye-tracking data. Based on
the established SJTU-TIS database, we analyze the influence of various text
descriptions on visual attention. Then, to facilitate the development of
saliency prediction models considering text influence, we construct a benchmark
for the established SJTU-TIS database using state-of-the-art saliency models.
Finally, considering the effect of text descriptions on visual attention, while
most existing saliency models ignore this impact, we further propose a
text-guided saliency (TGSal) prediction model, which extracts and integrates
both image features and text features to predict the image saliency under
various text-description conditions. Our proposed model significantly
outperforms the state-of-the-art saliency models on both the SJTU-TIS database
and the pure image saliency databases in terms of various evaluation metrics.
The SJTU-TIS database and the code of the proposed TGSal model will be released
at: https://github.com/IntMeGroup/TGSal.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要