Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Models
arxiv(2024)
Abstract
Social bias is shaped by the accumulation of social perceptions towards
targets across various demographic identities. To fully understand such social
bias in large language models (LLMs), it is essential to consider the composite
of social perceptions from diverse perspectives among identities. Previous
studies have either evaluated biases in LLMs by indirectly assessing the
presence of sentiments towards demographic identities in the generated text or
measuring the degree of alignment with given stereotypes. These methods have
limitations in directly quantifying social biases at the level of distinct
perspectives among identities. In this paper, we aim to investigate how social
perceptions from various viewpoints contribute to the development of social
bias in LLMs. To this end, we propose a novel strategy to intuitively quantify
these social perceptions and suggest metrics that can evaluate the social
biases within LLMs by aggregating diverse social perceptions. The experimental
results show the quantitative demonstration of the social attitude in LLMs by
examining social perception. The analysis we conducted shows that our proposed
metrics capture the multi-dimensional aspects of social bias, enabling a
fine-grained and comprehensive investigation of bias in LLMs.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined