AttributionBench: How Hard is Automatic Attribution Evaluation?
CoRR(2024)
摘要
Modern generative search engines enhance the reliability of large language
model (LLM) responses by providing cited evidence. However, evaluating the
answer's attribution, i.e., whether every claim within the generated responses
is fully supported by its cited evidence, remains an open problem. This
verification, traditionally dependent on costly human evaluation, underscores
the urgent need for automatic attribution evaluation methods. To bridge the gap
in the absence of standardized benchmarks for these methods, we present
AttributionBench, a comprehensive benchmark compiled from various existing
attribution datasets. Our extensive experiments on AttributionBench reveal the
challenges of automatic attribution evaluation, even for state-of-the-art LLMs.
Specifically, our findings show that even a fine-tuned GPT-3.5 only achieves
around 80
analysis of more than 300 error cases indicates that a majority of failures
stem from the model's inability to process nuanced information, and the
discrepancy between the information the model has access to and that human
annotators do.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要