Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method
arxiv(2023)
摘要
Large Language Models (LLMs) have shown great potential in Natural Language
Processing (NLP) tasks. However, recent literature reveals that LLMs generate
nonfactual responses intermittently, which impedes the LLMs' reliability for
further utilization. In this paper, we propose a novel self-detection method to
detect which questions that a LLM does not know that are prone to generate
nonfactual results. Specifically, we first diversify the textual expressions
for a given question and collect the corresponding answers. Then we examine the
divergencies between the generated answers to identify the questions that the
model may generate falsehoods. All of the above steps can be accomplished by
prompting the LLMs themselves without referring to any other external
resources. We conduct comprehensive experiments and demonstrate the
effectiveness of our method on recently released LLMs, e.g., Vicuna, ChatGPT,
and GPT-4.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要