Context-aware Decoding Reduces Hallucination in Query-focused Summarization
CoRR(2023)
摘要
Query-focused summarization (QFS) aims to provide a summary of a single
document/multi documents that can satisfy the information needs of a given
query. It is useful for various real-world applications, such as abstractive
snippet generation or more recent retrieval augmented generation (RAG). A
prototypical QFS pipeline consists of a retriever (sparse or dense retrieval)
and a generator (usually a large language model). However, applying large
language models (LLM) potentially leads to hallucinations, especially when the
evidence contradicts the prior belief of LLMs. There has been growing interest
in developing new decoding methods to improve generation quality and reduce
hallucination. In this work, we conduct a large-scale reproducibility study on
one recently proposed decoding method – Context-aware Decoding (CAD). In
addition to replicating CAD's experiments on news summarization datasets, we
include experiments on QFS datasets, and conduct more rigorous analysis on
computational complexity and hyperparameter sensitivity. Experiments with eight
different language models show that performance-wise, CAD improves QFS quality
by (1) reducing factuality errors/hallucinations while (2) mostly retaining the
match of lexical patterns, measured by ROUGE scores, while also at a cost of
increased inference-time FLOPs and reduced decoding speed. The code
implementation based on Huggingface Library is made available
https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要