CRAB: Assessing the Strength of Causal Relationships Between Real-world Events.
CoRR(2023)
摘要
Understanding narratives requires reasoning about the cause-and-effect
relationships between events mentioned in the text. While existing foundation
models yield impressive results in many NLP tasks requiring reasoning, it is
unclear whether they understand the complexity of the underlying network of
causal relationships of events in narratives. In this work, we present CRAB, a
new Causal Reasoning Assessment Benchmark designed to evaluate causal
understanding of events in real-world narratives. CRAB contains fine-grained,
contextual causality annotations for ~2.7K pairs of real-world events that
describe various newsworthy event timelines (e.g., the acquisition of Twitter
by Elon Musk). Using CRAB, we measure the performance of several large language
models, demonstrating that most systems achieve poor performance on the task.
Motivated by classical causal principles, we also analyze the causal structures
of groups of events in CRAB, and find that models perform worse on causal
reasoning when events are derived from complex causal structures compared to
simple linear causal chains. We make our dataset and code available to the
research community.
更多查看译文
关键词
causal relationships,events,crab,real-world
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要