TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling
CoRR(2024)
摘要
As a cross-modal task, visual storytelling aims to generate a story for an
ordered image sequence automatically. Different from the image captioning task,
visual storytelling requires not only modeling the relationships between
objects in the image but also mining the connections between adjacent images.
Recent approaches primarily utilize either end-to-end frameworks or multi-stage
frameworks to generate relevant stories, but they usually overlook latent topic
information. In this paper, in order to generate a more coherent and relevant
story, we propose a novel method, Topic Aware Reinforcement Network for VIsual
StoryTelling (TARN-VIST). In particular, we pre-extracted the topic information
of stories from both visual and linguistic perspectives. Then we apply two
topic-consistent reinforcement learning rewards to identify the discrepancy
between the generated story and the human-labeled story so as to refine the
whole generation process. Extensive experimental results on the VIST dataset
and human evaluation demonstrate that our proposed model outperforms most of
the competitive models across multiple evaluation metrics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要