LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization
CoRR(2024)
摘要
Pretrained language models (PLMs) have shown remarkable generalization toward
multiple tasks and languages. Nonetheless, the generalization of PLMs towards
unseen languages is poor, resulting in significantly worse language
performance, or even generating nonsensical responses that are comparable to a
random baseline. This limitation has been a longstanding problem of PLMs
raising the problem of diversity and equal access to language modeling
technology. In this work, we solve this limitation by introducing LinguAlchemy,
a regularization technique that incorporates various aspects of languages
covering typological, geographical, and phylogenetic constraining the resulting
representation of PLMs to better characterize the corresponding linguistics
constraints. LinguAlchemy significantly improves the accuracy performance of
mBERT and XLM-R on unseen languages by 18
fully finetuned models and displaying a high degree of unseen language
generalization. We further introduce AlchemyScale and AlchemyTune, extension of
LinguAlchemy which adjusts the linguistic regularization weights automatically,
alleviating the need for hyperparameter search. LinguAlchemy enables better
cross-lingual generalization to unseen languages which is vital for better
inclusivity and accessibility of PLMs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要