The Real Cost of AI-Generated Code: It's Not All It's Cracked Up To Be

There is a lot of hype surrounding the use of GenAI for Development. “Anthropic CEO Dario Amodei said earlier this week that AI will write 90% of the code for software engineers within the next three to six months and every line of code within the next year.”[1]

I’ve been working in the software industry for nearly forty years. When I hear big promises about how a technology will revolutionize the world (e.g., Crypto, 4GLs, Digital Audio Tape, Smell-o-Vision, Segway), I want to see what’s happening on the ground. Years of work in the Agile world have taught me to engage in Systems Thinking and ask: what are the second and third order effects of what I see in front of me? I also want to understand if the claims I see are well-founded. Under what conditions are they true?

In this article, I will focus on the effects of GenAI on the quality and maintainability of our code bases. In subsequent articles, I will examine their impact on the people who use them and the promise of increased productivity.

Bold Claims

In “Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle”[2], GitHub CEO Thomas Dohmke claims that the AI-powered tool Copilot is helping developers write code faster. The research has been submitted to a reputable journal and will undergo peer review. Reportedy, developers:

• Accepted 30% of the suggestions that Copilot made
• Completed tasks 55% faster
• Wrote 46% more code
• 75% felt more fulfilled

Here’s the problem with those claims. The core of the paper assumes that all code added to a repository is equally valuable, but that assumption is invalid. The core claim of the paper is based on the age-old belief that more lines of code are better, and that is a flawed premise.

More code != more value:

• Some (many ;-)) changes introduce defects.
• Some features are poorly understood, and so the Copilot built the wrong thing.
• More code increases the attack surface and makes it easier for an attacker to break into the system.
• More code increases the complexity of the system, making it harder to add new features later.

In this case, more code appears mostly to equate to more Technical Debt. As the debt increases we end up with a big ball of mud that has other adverse effects:

• It takes from 2x to 10x as long to add new features as the code becomes a mess.
• Changes or new feature work in messy code is 15 times more likely to have defects.

(See CodeScene: The Business Impact of Code Quality[3]. Key takeaway - messy code is more expensive for your business.)

Reality Check

GitClear conducted a thorough analysis of data[4] based on the users of their “Software Intelligence System” and they saw: an increase in the volume of code added; an increase in churn; an increase in copy and pasted code; a decrease in the amount of code moved.

They define churn as code added, changed or removed in a 2-week period. Churn is often code that was written, committed, and later found to be defective. This is not a number we want to see go up. Before 2022, Churn was around 3-4%. In the years since, Churn has grown at a rate proportional to the uptake of Copilot/ChatGPT usage. (Caveat, they’ve demonstrated a correlation and not causation - the correlation coefficient is 0.98.)

Technical debt is also increasing because there is more copy and paste code (the LLM didn’t notice it was a duplicate) and a reduction in refactoring (less code moved).

All told, GitClear’s analysis suggests that Copilot didn’t add $1.5 trillion to the economy. If anything, the users have spent money from the future economy for no short-term gain. This analysis compared 2023 to prior years, so it is based on a period before the largest growth period for LLM use. I expect this understates the problems we will see in 2024 and beyond.

Summary

I’m not saying that GenAI can’t help; I think we will get there. With prototyping and exploratory work, GenAI is already making possible things that didn’t exist previously. However, current GenAI use often creates bigger problems in the long term.

I’ve said this several times: the Theory of Constraints tells us that if something isn’t the bottleneck, don’t optimize for it. Don’t optimize for individual productivity; optimize for the team. Measure how long it takes to deliver value to the customer (probably Cycle Time). Use LLM-based tools where they help you with that, not because the CEO of an AI company promises more code faster.