Quantifying the Persona Effect in LLM Simulations
CoRR(2024)
摘要
Large language models (LLMs) have shown remarkable promise in simulating
human language use and behavior. In this study, we delve into the intersection
of persona variables and the capability of LLMs to simulate different
perspectives. We find that persona variables can explain <10% variance in
annotations in existing subjective NLP datasets. Nonetheless, incorporating
them via prompting in LLMs provides modest improvement. Persona prompting is
most effective on data samples where disagreements among annotators are
frequent yet confined to a limited range. A linear correlation exists: the more
persona variables influence human annotations, the better LLMs predictions are
using persona prompting. However, when the utility of persona variables is low
(i.e., explaining <10% of human annotations), persona prompting has little
effect. Most subjective NLP datasets fall into this category, casting doubt on
simulating diverse perspectives in the current NLP landscape.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要