Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs
arxiv(2024)
摘要
Benchmarking is the de-facto standard for evaluating LLMs, due to its speed,
replicability and low cost. However, recent work has pointed out that the
majority of the open source benchmarks available today have been contaminated
or leaked into LLMs, meaning that LLMs have access to test data during
pretraining and/or fine-tuning. This raises serious concerns about the validity
of benchmarking studies conducted so far and the future of evaluation using
benchmarks. To solve this problem, we propose Private Benchmarking, a solution
where test datasets are kept private and models are evaluated without revealing
the test data to the model. We describe various scenarios (depending on the
trust placed on model owners or dataset owners), and present solutions to avoid
data contamination using private benchmarking. For scenarios where the model
weights need to be kept private, we describe solutions from confidential
computing and cryptography that can aid in private benchmarking. Finally, we
present solutions the problem of benchmark dataset auditing, to ensure that
private benchmarks are of sufficiently high quality.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要