RELIC: Investigating Large Language Model Responses Using Self-Consistency.

Large Language Models (LLMs) are notorious for blending fact with fiction andgenerating non-factual content, known as hallucinations. To address thischallenge, we propose an interactive system that helps users gain insight intothe reliability of the generated text. Our approach is based on the idea thatthe self-consistency of multiple samples generated by the same LLM relates toits confidence in individual claims in the generated texts. Using this idea, wedesign RELIC, an interactive system that enables users to investigate andverify semantic-level variations in multiple long-form responses. This allowsusers to recognize potentially inaccurate information in the generated text andmake necessary corrections. From a user study with ten participants, wedemonstrate that our approach helps users better verify the reliability of thegenerated text. We further summarize the design implications and lessonslearned from this research for future studies of reliable human-LLMinteractions.