Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants
International Conference on Computational Linguistics(2024)
摘要
Recent works demonstrate that voice assistants do not perform equally well
for everyone, but research on demographic robustness of speech technologies is
still scarce. This is mainly due to the rarity of large datasets with
controlled demographic tags. This paper introduces the Sonos Voice Control Bias
Assessment Dataset, an open dataset composed of voice assistant requests for
North American English in the music domain (1,038 speakers, 166 hours, 170k
audio samples, with 9,040 unique labelled transcripts) with a controlled
demographic diversity (gender, age, dialectal region and ethnicity). We also
release a statistical demographic bias assessment methodology, at the
univariate and multivariate levels, tailored to this specific use case and
leveraging spoken language understanding metrics rather than transcription
accuracy, which we believe is a better proxy for user experience. To
demonstrate the capabilities of this dataset and statistical method to detect
demographic bias, we consider a pair of state-of-the-art Automatic Speech
Recognition and Spoken Language Understanding models. Results show
statistically significant differences in performance across age, dialectal
region and ethnicity. Multivariate tests are crucial to shed light on mixed
effects between dialectal region, gender and age.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要