I Could've Asked That: Reformulating Unanswerable Questions
arxiv(2024)
摘要
When seeking information from unfamiliar documents, users frequently pose
questions that cannot be answered by the documents. While existing large
language models (LLMs) identify these unanswerable questions, they do not
assist users in reformulating their questions, thereby reducing their overall
utility. We curate CouldAsk, an evaluation benchmark composed of existing and
new datasets for document-grounded question answering, specifically designed to
study reformulating unanswerable questions. We evaluate state-of-the-art
open-source and proprietary LLMs on CouldAsk. The results demonstrate the
limited capabilities of these models in reformulating questions. Specifically,
GPT-4 and Llama2-7B successfully reformulate questions only 26
time, respectively. Error analysis shows that 62
reformulations stem from the models merely rephrasing the questions or even
generating identical questions. We publicly release the benchmark and the code
to reproduce the experiments.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要