Understanding the Effect of Model Compression on Social Bias in Large Language Models.
EMNLP 2023(2023)
摘要
Large Language Models (LLMs) trained with self-supervision on vast corpora of
web text fit to the social biases of that text. Without intervention, these
social biases persist in the model's predictions in downstream tasks, leading
to representational harm. Many strategies have been proposed to mitigate the
effects of inappropriate social biases learned during pretraining.
Simultaneously, methods for model compression have become increasingly popular
to reduce the computational burden of LLMs. Despite the popularity and need for
both approaches, little work has been done to explore the interplay between
these two. We perform a carefully controlled study of the impact of model
compression via quantization and knowledge distillation on measures of social
bias in LLMs. Longer pretraining and larger models led to higher social bias,
and quantization showed a regularizer effect with its best trade-off around 20%
of the original pretraining time.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要