Understanding the Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation
CoRR(2024)
摘要
Pre-trained language models (LMs) are able to perform complex reasoning
without explicit fine-tuning. To understand how pre-training with a next-token
prediction objective contributes to the emergence of such reasoning capability,
we propose that we can view an LM as deriving new conclusions by aggregating
indirect reasoning paths seen at pre-training time. We found this perspective
effective in two important cases of reasoning: logic reasoning with knowledge
graphs (KGs) and math reasoning with math word problems (MWPs). More
specifically, we formalize the reasoning paths as random walk paths on the
knowledge/reasoning graphs. Analyses of learned LM distributions suggest that a
weighted sum of relevant random walk path probabilities is a reasonable way to
explain how LMs reason. Experiments and analysis on multiple KG and MWP
datasets reveal the effect of training on random walk paths and suggest that
augmenting unlabeled random walk reasoning paths can improve real-world
multi-step reasoning performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要