Distilling Mathematical Reasoning Capabilities into Small Language Models
arxiv(2024)
摘要
This work addresses the challenge of democratizing advanced Large Language
Models (LLMs) by compressing their mathematical reasoning capabilities into
sub-billion parameter Small Language Models (SLMs) without compromising
performance. We introduce Equation-of-Thought Distillation (EoTD), a novel
technique that encapsulates the reasoning process into equation-based
representations to construct an EoTD dataset for fine-tuning SLMs.
Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to
enhance the reasoning performance of SLMs. This involves creating a reasoning
dataset with multiple thought processes, including Chain-of-Thought (CoT),
Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for
fine-tuning. Our experimental findings demonstrate that EoTD significantly
boosts the reasoning abilities of SLMs, while ETD enables these models to
achieve state-of-the-art reasoning performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要