Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision
CoRR(2024)
摘要
Process supervision, using a trained verifier to evaluate the intermediate
steps generated by reasoner, has demonstrated significant improvements in
multi-step problem solving. In this paper, to avoid expensive human annotation
effort on the verifier training data, we introduce Model-induced Process
Supervision (MiPS), a novel method for automating data curation. MiPS annotates
an intermediate step by sampling completions of this solution through the
reasoning model, and obtaining an accuracy defined as the proportion of correct
completions. Errors in the reasoner would cause MiPS to underestimate the
accuracy of intermediate steps, therefore, we suggest and empirically show that
verification focusing on high predicted scores of the verifier shall be
preferred over that of low predicted scores, contrary to prior work. Our
approach significantly improves the performance of PaLM 2 on math and coding
tasks (accuracy +0.67
an output supervision trained verifier). Additionally, our study demonstrates
that the verifier exhibits strong generalization ability across different
reasoning models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要