Code written by OpenAI and praised by GitHub may not be as good as Github says
- GitHub says its AI-generated code is more readable, reliable, and maintainable
- The test focused on a highly repetitive task: the ultimate role of AI
- Only 243 developers participated in the survey
Software developer Dan Cîmpianu has criticized the quality of AI-generated code in a blog post targeting GitHub’s claims about its Copilot AI tool.
More specifically, the Romanian developer recently described the statistical accuracy and experimental design used by GitHub studywhere it claimed that Copilot-supported code was “significantly more functional, readable, reliable, maintainable, and concise.”
However, the research focused on writing API endpoints for a web server, or CRUDs (Create, Read, Update and Delete actions), which Cîmpianu described as “one of the most boring, repetitive, uninspired and cognitively unchallenged aspects of development. ”
Is GitHub’s AI code actually that good?
The study compared GitHub’s OpenAI-powered AI-generated code with that of more than 200 experienced developers and found that the AI code performed better across multiple metrics.
However, Cîmpianu has criticized GitHub for using percentages to indicate differences without actually providing the basic data for comparison, which could make the percentage values appear artificially higher than they are.
GitHub’s research also defines bugs as “inconsistent naming, unclear identifiers, excessive line length, excessive white space, missing documentation, repetitive code, excessive branching or loop depth, insufficient separation of functionality, and variable complexity,” meaning bugs introduced by the code produced were removed. not included in the statistics
Another criticism of the study is that despite being “a home to 1 billion developers,” the study only uses a sample size of 243 developers.
Cîmpianu concluded: “This doesn’t even seem to be an attempt [be] aimed at developers, but rather has the scent of marketing, aimed at the C-suites with purchasing power.”
Furthermore, the developer also highlighted the skills needed to write strong code, stating that AI should be seen as a complement and a tool and not a replacement for advanced training.
