RELIC: Investigating Large Language Model Responses Using Self-Consistency.
CHI '24 Proceedings of the CHI Conference on Human Factors in Computing Systems(2024)
Abstract
Large Language Models (LLMs) are notorious for blending fact with fiction andgenerating non-factual content, known as hallucinations. To address thischallenge, we propose an interactive system that helps users gain insight intothe reliability of the generated text. Our approach is based on the idea thatthe self-consistency of multiple samples generated by the same LLM relates toits confidence in individual claims in the generated texts. Using this idea, wedesign RELIC, an interactive system that enables users to investigate andverify semantic-level variations in multiple long-form responses. This allowsusers to recognize potentially inaccurate information in the generated text andmake necessary corrections. From a user study with ten participants, wedemonstrate that our approach helps users better verify the reliability of thegenerated text. We further summarize the design implications and lessonslearned from this research for future studies of reliable human-LLMinteractions.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined