CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models
CoRR(2024)
摘要
This paper introduces the "CIVICS: Culturally-Informed Values-Inclusive
Corpus for Societal impacts" dataset, designed to evaluate the social and
cultural variation of Large Language Models (LLMs) across multiple languages
and value-sensitive topics. We create a hand-crafted, multilingual dataset of
value-laden prompts which address specific socially sensitive topics, including
LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
CIVICS is designed to generate responses showing LLMs' encoded and implicit
values. Through our dynamic annotation processes, tailored prompt design, and
experiments, we investigate how open-weight LLMs respond to value-sensitive
issues, exploring their behavior across diverse linguistic and cultural
contexts. Using two experimental set-ups based on log-probabilities and
long-form responses, we show social and cultural variability across different
LLMs. Specifically, experiments involving long-form responses demonstrate that
refusals are triggered disparately across models, but consistently and more
frequently in English or translated statements. Moreover, specific topics and
sources lead to more pronounced differences across model answers, particularly
on immigration, LGBTQI rights, and social welfare. As shown by our experiments,
the CIVICS dataset aims to serve as a tool for future research, promoting
reproducibility and transparency across broader linguistic settings, and
furthering the development of AI technologies that respect and reflect global
cultural diversities and value pluralism. The CIVICS dataset and tools will be
made available upon publication under open licenses; an anonymized version is
currently available at https://huggingface.co/CIVICS-dataset.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要