SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
ICLR 2024(2024)
摘要
We study the problem of procedure planning in instructional videos, which
aims to make a goal-oriented sequence of action steps given partial visual
state observations. The motivation of this problem is to learn a structured and
plannable state and action space. Recent works succeeded in sequence modeling
of steps with only sequence-level annotations accessible during training, which
overlooked the roles of states in the procedures. In this work, we point out
that State CHangEs MAtter (SCHEMA) for procedure planning in instructional
videos. We aim to establish a more structured state space by investigating the
causal relations between steps and states in procedures. Specifically, we
explicitly represent each step as state changes and track the state changes in
procedures. For step representation, we leveraged the commonsense knowledge in
large language models (LLMs) to describe the state changes of steps via our
designed chain-of-thought prompting. For state change tracking, we align visual
state observations with language state descriptions via cross-modal contrastive
learning, and explicitly model the intermediate states of the procedure using
LLM-generated state descriptions. Experiments on CrossTask, COIN, and NIV
benchmark datasets demonstrate that our proposed SCHEMA model achieves
state-of-the-art performance and obtains explainable visualizations.
更多查看译文
关键词
procedure planning,instructional video,state changes,large language models,vision and language alignment
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要