Towards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation
CoRR(2024)
摘要
Large language models have consistently struggled with complex reasoning
tasks, such as mathematical problem-solving. Investigating the internal
reasoning mechanisms of these models can help us design better model
architectures and training strategies, ultimately enhancing their reasoning
capabilities. In this study, we examine the matching mechanism employed by
Transformer for multi-step reasoning on a constructed dataset. We investigate
factors that influence the model's matching mechanism and discover that small
initialization and post-LayerNorm can facilitate the formation of the matching
mechanism, thereby enhancing the model's reasoning ability. Moreover, we
propose a method to improve the model's reasoning capability by adding
orthogonal noise. Finally, we investigate the parallel reasoning mechanism of
Transformers and propose a conjecture on the upper bound of the model's
reasoning ability based on this phenomenon. These insights contribute to a
deeper understanding of the reasoning processes in large language models and
guide designing more effective reasoning architectures and training strategies.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要