Where Are Large Language Models for Code Generation on GitHub?
arxiv(2024)
摘要
The increasing use of Large Language Models (LLMs) in software development
has garnered significant attention from researchers assessing the quality of
the code they generate. However, much of the research focuses on controlled
datasets such as HumanEval, which fail to adequately represent how developers
actually utilize LLMs' code generation capabilities or clarify the
characteristics of LLM-generated code in real-world development scenarios. To
bridge this gap, our study investigates the characteristics of LLM-generated
code and its corresponding projects hosted on GitHub. Our findings reveal
several key insights: (1) ChatGPT and Copilot are the most frequently utilized
for generating code on GitHub. In contrast, there is very little code generated
by other LLMs on GitHub. (2) Projects containing ChatGPT/Copilot-generated code
are often small and less known, led by individuals or small teams. Despite
this, most projects are continuously evolving and improving. (3)
ChatGPT/Copilot is mainly utilized for generating Python, Java, and TypeScript
scripts for data processing and transformation. C/C++ and JavaScript code
generation focuses on algorithm and data structure implementation and user
interface code. Most ChatGPT/Copilot-generated code snippets are relatively
short and exhibit low complexity. (4) Compared to human-written code,
ChatGPT/Copilot-generated code exists in a small proportion of projects and
generally undergoes fewer modifications. Additionally, modifications due to
bugs are even fewer, ranging from just 3
Most comments on ChatGPT/Copilot-generated code lack detailed information,
often only stating the code's origin without mentioning prompts, human
modifications, or testing status. Based on these findings, we discuss the
implications for researchers and practitioners.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要