AlignBench: Benchmarking Chinese Alignment of Large Language Models
CoRR(2023)
摘要
Alignment has become a critical step for instruction-tuned Large Language
Models (LLMs) to become helpful assistants. However, effective evaluation of
alignment for emerging Chinese LLMs is still significantly lacking, calling for
real-scenario grounded, open-ended, challenging and automatic evaluations
tailored for alignment. To fill in this gap, we introduce AlignBench, a
comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in
Chinese. Equipped with a human-in-the-loop data curation pipeline, our
benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge with
Chain-of-Thought to generate explanations and final ratings as evaluations,
ensuring high reliability and interpretability. Furthermore, we developed a
dedicated companion evaluator LLM -- CritiqueLLM, which recovers 95\% of
GPT-4's evaluation ability and will be provided via public APIs to researchers
for evaluation of alignment in Chinese LLMs. All evaluation codes, data, and
LLM generations are available at \url{https://github.com/THUDM/AlignBench}.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要