Changing Answer Order Can Decrease MMLU Accuracy
arxiv(2024)
摘要
As large language models (LLMs) have grown in prevalence, particular
benchmarks have become essential for the evaluation of these models and for
understanding model capabilities. Most commonly, we use test accuracy averaged
across multiple subtasks in order to rank models on leaderboards, to determine
which model is best for our purposes. In this paper, we investigate the
robustness of the accuracy measurement on a widely used multiple choice
question answering dataset, MMLU. When shuffling the answer label contents, we
find that all explored models decrease in accuracy on MMLU, but not every model
is equally sensitive. These findings suggest a possible adjustment to the
standard practice of leaderboard testing, where we additionally consider the
percentage of examples each model answers correctly by random chance.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要