Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot
CoRR(2024)
摘要
GitHub Copilot is an AI-enabled tool that automates program synthesis. It has
gained significant attention since its launch in 2021. Recent studies have
extensively examined Copilot's capabilities in various programming tasks, as
well as its security issues. However, little is known about the effect of
different natural languages on code suggestion. Natural language is considered
a social bias in the field of NLP, and this bias could impact the diversity of
software engineering. To address this gap, we conducted an empirical study to
investigate the effect of three popular natural languages (English, Japanese,
and Chinese) on Copilot. We used 756 questions of varying difficulty levels
from AtCoder contests for evaluation purposes. The results highlight that the
capability varies across natural languages, with Chinese achieving the worst
performance. Furthermore, regardless of the type of natural language, the
performance decreases significantly as the difficulty of questions increases.
Our work represents the initial step in comprehending the significance of
natural languages in Copilot's capability and introduces promising
opportunities for future endeavors.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要