Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis
arxiv(2023)
Abstract
Warning: this paper contains content that may be offensive or upsetting.
Most hate speech datasets neglect the cultural diversity within a single
language, resulting in a critical shortcoming in hate speech detection. To
address this, we introduce CREHate, a CRoss-cultural English Hate speech
dataset. To construct CREHate, we follow a two-step procedure: 1) cultural post
collection and 2) cross-cultural annotation. We sample posts from the SBIC
dataset, which predominantly represents North America, and collect posts from
four geographically diverse English-speaking countries (Australia, United
Kingdom, Singapore, and South Africa) using culturally hateful keywords we
retrieve from our survey. Annotations are collected from the four countries
plus the United States to establish representative labels for each country. Our
analysis highlights statistically significant disparities across countries in
hate speech annotations. Only 56.2
among all countries, with the highest pairwise label difference rate of 26
Qualitative analysis shows that label disagreement occurs mostly due to
different interpretations of sarcasm and the personal bias of annotators on
divisive topics. Lastly, we evaluate large language models (LLMs) under a
zero-shot setting and show that current LLMs tend to show higher accuracies on
Anglosphere country labels in CREHate. Our dataset and codes are available at:
https://github.com/nlee0212/CREHate
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined