FanOutQA: Multi-Hop, Multi-Document Question Answering for Large Language Models
CoRR(2024)
摘要
One type of question that is commonly found in day-to-day scenarios is
“fan-out” questions, complex multi-hop, multi-document reasoning questions
that require finding information about a large number of entities. However,
there exist few resources to evaluate this type of question-answering
capability among large language models. To evaluate complex reasoning in LLMs
more fully, we present FanOutQA, a high-quality dataset of fan-out
question-answer pairs and human-annotated decompositions with English Wikipedia
as the knowledge base. We formulate three benchmark settings across our dataset
and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B,
finding that contemporary models still have room to improve reasoning over
inter-document dependencies in a long context. We provide our dataset and
open-source tools to run models to encourage evaluation at https://fanoutqa.com
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要