Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository
CoRR(2024)
Abstract
LLMs have demonstrated significant potential in code generation tasks,
achieving promising results at the function or statement level in various
benchmarks. However, the complexities associated with creating code artifacts
like classes, particularly within the context of real-world software
repositories, remain underexplored. Existing research often treats class-level
generation as an isolated task, neglecting the intricate dependencies and
interactions that characterize real-world software development environments. To
address this gap, we introduce RepoClassBench, a benchmark designed to
rigorously evaluate LLMs in generating complex, class-level code within
real-world repositories. RepoClassBench includes natural language to class
generation tasks across Java and Python, from a selection of public
repositories. We ensure that each class in our dataset not only has cross-file
dependencies within the repository but also includes corresponding test cases
to verify its functionality. We find that current models struggle with the
realistic challenges posed by our benchmark, primarily due to their limited
exposure to relevant repository contexts. To address this shortcoming, we
introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs
with static analysis tools to iteratively navigate reason about
repository-level context in an agent-based framework. Our experiments
demonstrate that RRR significantly outperforms existing baselines on
RepoClassBench, showcasing its effectiveness across programming languages and
in various settings. Our findings emphasize the need for benchmarks that
incorporate repository-level dependencies to more accurately reflect the
complexities of software development. Our work illustrates the benefits of
leveraging specialized tools to enhance LLMs understanding of repository
context. We plan to make our dataset and evaluation harness public.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined