When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives
arxiv(2024)
Abstract
Reasoning is most powerful when an LLM accurately aggregates relevant
information. We examine the critical role of information aggregation in
reasoning by requiring the LLM to analyze sports narratives. To succeed at this
task, an LLM must infer points from actions, identify related entities,
attribute points accurately to players and teams, and compile key statistics to
draw conclusions. We conduct comprehensive experiments with real NBA basketball
data and present SportsGen, a new method to synthesize game narratives. By
synthesizing data, we can rigorously evaluate LLMs' reasoning capabilities
under complex scenarios with varying narrative lengths and density of
information. Our findings show that most models, including GPT-4o, often fail
to accurately aggregate basketball scores due to frequent scoring patterns.
Open-source models like Llama-3 further suffer from significant score
hallucinations. Finally, the effectiveness of reasoning is influenced by
narrative complexity, information density, and domain-specific terms,
highlighting the challenges in analytical reasoning tasks.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined