A Study on the Calibration of In-context Learning
CoRR(2023)
摘要
Accurate uncertainty quantification is crucial for the safe deployment of
language models (LMs), and prior research has demonstrated improvements in the
calibration of modern LMs. Our study focuses on in-context learning (ICL), a
prevalent method for adapting static LMs through tailored prompts, and examines
the balance between performance and calibration across a broad spectrum of
natural language understanding and reasoning tasks. Through comprehensive
experiments, we observe that, with an increasing number of ICL examples, models
initially exhibit increased miscalibration before achieving better calibration
and miscalibration tends to arise in low-shot settings. Moreover, we find that
methods aimed at improving usability, such as fine-tuning and chain-of-thought
(CoT) prompting, can lead to miscalibration and unreliable natural language
explanations, suggesting that new methods may be required for scenarios where
models are expected to be reliable.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要