MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
CoRR(2024)
Abstract
Recent advancements in large language models (LLMs) have greatly improved
code generation, specifically at the function level. For instance, GPT-4 has
achieved an 88.4
adequacy of existing benchmarks in thoroughly assessing function-level code
generation capabilities. Our study analyzed two common benchmarks, HumanEval
and MBPP, and found that these might not thoroughly evaluate LLMs' code
generation capacities due to limitations in quality, difficulty, and
granularity. To resolve this, we introduce the Mostly Hard Python Problems
(MHPP) dataset, consisting of 140 unique human-curated problems. By focusing on
the combination of natural language and code reasoning, MHPP gauges LLMs'
abilities to comprehend specifications and restrictions, engage in multi-step
reasoning, and apply coding knowledge effectively. Initial evaluations of 22
LLMs using MHPP showed many high-performing models on HumanEval failed to
achieve similar success on MHPP. Moreover, MHPP highlighted various previously
undiscovered limitations within various LLMs, leading us to believe that it
could pave the way for a better understanding of LLMs' capabilities and
limitations. Dataset and code are available at
https://github.com/SparksofAGI/MHPP.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined