MCC-KD: Multi-CoT Consistent Knowledge Distillation.
CoRR(2023)
摘要
Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting.~Recently, there has been a growing interest in transferring these reasoning abilities from LLMs to smaller models.~However, achieving both the diversity and consistency in rationales presents a challenge.~In this paper, we focus on enhancing these two aspects and propose Multi-CoT Consistent Knowledge Distillation (MCC-KD) to efficiently distill the reasoning capabilities. In MCC-KD, we generate multiple rationales for each question and enforce consistency among the corresponding predictions by minimizing the bidirectional KL-divergence between the answer distributions.~We investigate the effectiveness of MCC-KD with different model architectures (LLaMA/FlanT5) and various model scales (3B/7B/11B/13B) on both mathematical reasoning and commonsense reasoning benchmarks. The empirical results not only confirm MCC-KD's superior performance on in-distribution datasets but also highlight its robust generalization ability on out-of-distribution datasets.
更多查看译文
关键词
knowledge,multi-cot
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要