Confidence under the Hood: an Investigation into the Confidence-Probability Alignment in Large Language Models

As the use of Large Language Models (LLMs) becomes more widespread,understanding their self-evaluation of confidence in generated responsesbecomes increasingly important as it is integral to the reliability of theoutput of these models. We introduce the concept of Confidence-ProbabilityAlignment, that connects an LLM's internal confidence, quantified by tokenprobabilities, to the confidence conveyed in the model's response whenexplicitly asked about its certainty. Using various datasets and promptingtechniques that encourage model introspection, we probe the alignment betweenmodels' internal and expressed confidence. These techniques encompass usingstructured evaluation scales to rate confidence, including answer options whenprompting, and eliciting the model's confidence level for outputs it does notrecognize as its own. Notably, among the models analyzed, OpenAI's GPT-4 showedthe strongest confidence-probability alignment, with an average Spearman'sρ̂ of 0.42, across a wide range of tasks. Our work contributes to theongoing efforts to facilitate risk assessment in the application of LLMs and tofurther our understanding of model trustworthiness.