Reliable, Adaptable, and Attributable Language Models with Retrieval
arxiv(2024)
摘要
Parametric language models (LMs), which are trained on vast amounts of web
data, exhibit remarkable flexibility and capability. However, they still face
practical challenges such as hallucinations, difficulty in adapting to new data
distributions, and a lack of verifiability. In this position paper, we advocate
for retrieval-augmented LMs to replace parametric LMs as the next generation of
LMs. By incorporating large-scale datastores during inference,
retrieval-augmented LMs can be more reliable, adaptable, and attributable.
Despite their potential, retrieval-augmented LMs have yet to be widely adopted
due to several obstacles: specifically, current retrieval-augmented LMs
struggle to leverage helpful text beyond knowledge-intensive tasks such as
question answering, have limited interaction between retrieval and LM
components, and lack the infrastructure for scaling. To address these, we
propose a roadmap for developing general-purpose retrieval-augmented LMs. This
involves a reconsideration of datastores and retrievers, the exploration of
pipelines with improved retriever-LM interaction, and significant investment in
infrastructure for efficient training and inference.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要