Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models
CoRR(2024)
摘要
Can large multimodal models have a human-like ability for emotional and
social reasoning, and if so, how does it work? Recent research has discovered
emergent theory-of-mind (ToM) reasoning capabilities in large language models
(LLMs). LLMs can reason about people's mental states by solving various
text-based ToM tasks that ask questions about the actors' ToM (e.g., human
belief, desire, intention). However, human reasoning in the wild is often
grounded in dynamic scenes across time. Thus, we consider videos a new medium
for examining spatio-temporal ToM reasoning ability. Specifically, we ask
explicit probing questions about videos with abundant social and emotional
reasoning content. We develop a pipeline for multimodal LLM for ToM reasoning
using video and text. We also enable explicit ToM reasoning by retrieving key
frames for answering a ToM question, which reveals how multimodal LLMs reason
about ToM.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要