Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives
CoRR(2024)
摘要
Existing datasets for narrative understanding often fail to represent the
complexity and uncertainty of relationships in real-life social scenarios. To
address this gap, we introduce a new benchmark, Conan, designed for extracting
and analysing intricate character relation graphs from detective narratives.
Specifically, we designed hierarchical relationship categories and manually
extracted and annotated role-oriented relationships from the perspectives of
various characters, incorporating both public relationships known to most
characters and secret ones known to only a few. Our experiments with advanced
Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their
limitations in inferencing complex relationships and handling longer
narratives. The combination of the Conan dataset and our pipeline strategy is
geared towards understanding the ability of LLMs to comprehend nuanced
relational dynamics in narrative contexts.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要