How far can we push AI autonomy in code generation?

Testing the Limits of Autonomous Code Generation

What would it take for a large language model to generate an entire working application from a natural-language specification — no human coding, testing or deployment in between? To gauge where the technology stands today, we built an agentic workflow and pointed it at a deliberately simple target: a Spring Boot CRUD API with persistence, services, controllers, and tests at every layer.

We ran the generation with Claude-Sonnet models (3.7 and 4), which we found to be the strongest available LLMs for coding tasks. The setup sat on top of the Roo Code / Kilo Code coding assistant, chosen because it can orchestrate subtasks in separate context windows — a requirement for simulating a multi-agent setup without building one from scratch. With an allow-list of terminal commands, the only human interaction needed was an occasional approval click.

The experiment produced working applications, but the path there revealed persistent gaps in AI autonomy. The models invented unrequested features, made shifting assumptions about ambiguous requirements, and reported success even when tests were failing. As requirements grew more complex, the issues compounded. The conclusion: reusable prompts and reference applications meaningfully improve AI-assisted development, but a human supervisor remains essential.

Strategies for Taming the Generator

We applied a series of techniques one at a time to see which ones improved reliability, code quality, and the repeatability of the generation. Each was an attempt to add control to a fundamentally probabilistic process.

Choosing a Familiar Stack

Spring Boot was a deliberate choice from the start. It is a common framework that appears extensively in LLM training data, it handles much of the boilerplate itself so there is less code for the AI to produce, and its layered architecture — Controller -> Service -> Repository -> Entity — follows well-established patterns that are easy to communicate to a model.

Separating Concerns Across Agents

Instead of one long generation session, we split the process into roles: requirements analyst, bootstrapper, backend designer, and generators for each persistence, service, and controller layer, followed by an E2E tester and a code reviewer. Each step ran in its own LLM session with its own role and instruction set.

The reasoning is twofold. First, LLM output degrades as sessions lengthen, even with growing context windows. Short, focused sessions tend to produce better results. Second, assigning specific roles and perspectives to a model is an established way to improve output quality, and a multi-agent setup makes that natural.

Embracing Stack-Specific Prompts

We did not ask the model to "write a Spring Boot application" and hope for the best. Our prompts were explicitly tailored to the Spring target stack and its typical layer boundaries. General-purpose generation may be the promise of AI coding, but without detailed guidance, the output lacks the quality and context that real-world digital products demand. We wanted to see how stack-specific the instructions would need to become for both quality and repeatability.

Leaving Bootstrapping to Scripts

For the initial application skeleton, we used a deterministic shell script backed by the Spring Boot CLI rather than having the LLM guess at project structure. It is worth remembering that an agentic workflow does not need to be entirely AI-driven; mixing in conventional tooling where it works better is still an option.

Showing, Not Just Telling, with Code Examples

The single most effective strategy was including concrete code snippets for each pattern — Entity, Repository, and so on. This mattered for more than style. Without prompting, the model defaulted to the outdated javax.persistence namespace instead of the current jakarta.persistence. For any engineering organization with a curated set of libraries and idioms, sample code is the clearest way to communicate those conventions and keep generated code consistent.

This also matters for maintenance. An AI that will update an application over time needs to adopt new framework versions as they arrive, not wait until they dominate the training data. Reliable example-driven prompting is a step toward that.

Anchoring with a Reference Application

Maintaining examples inside natural-language prompts proved tedious; there was no fast feedback on whether the snippets compiled or matched each other. To fix this, we created a reference application and exposed its code to the agent through a Model Context Protocol (MCP) server. This kept the samples compilable and mutually consistent without constant prompt editing.

Adding a Review Loop

A dedicated review agent checked the generated code against the original instructions, catching mistakes that the generating agents missed. An LLM's first pass rarely follows every instruction, particularly under a heavy prompt load. But when asked to compare its output against the spec, the model is generally good at spotting its own deviations and correcting them. The review step acted as a safety net for exactly that.

Modularizing the Codebase

We instructed the requirements-and-design agents to group the domain into aggregates and use those to define the package structure. Unlike most of the other strategies, this was hard to get right without human correction — it is a concept that challenges human developers too.

          An aggregate is a cluster of domain objects that can be treated as a
          single unit, it must stay internally consistent after each business
          operation.

          For each aggregate:
          - Name root and contained entities
          - Explain why this aggregate is sized the way it is
          (transaction size, concurrency, read/write patterns).

We did not spend much effort tuning those instructions, and it shows. But the goal was structural, not cosmetic. Effective modularization means a change touches fewer places, so the LLM needs less context to make it correctly. It also allows an agentic workflow to be re-run per module, lowering token usage and shrinking change sets, and it makes it easier to "freeze" unrelated modules against unintended edits.

Where the Strategy Set Falls Short

Even with all these controls in place, the workflow still showed clear limits. It would add features that were never requested, and the models flip-flopped on their own assumptions when requirements were left open. More troubling was the tendency to declare victory while tests were actually failing. The gap between "the code exists" and "the code works" is one that current models do not reliably recognize on their own.

The Verdict

These experiments were not about whether an LLM can write a small application — given a good prompt, it usually can. They were about how far autonomy can go right now, and where the ceiling is. The strategies that helped — especially code examples and the reference application — are lessons that apply directly to human-supervised AI-assisted development. But the failures around unrequested features, shifting assumptions, and false-sense-of-success reporting make a strong case that the human in the loop is not a temporary inconvenience. It is the current architecture.

Rounds three to five: scaling up the target

Testing the workflow against a modest target — roughly 15 iterations with domains like “Simple product catalog” or a library book tracker, capped at 3-5 entities and CRUD-only logic — produced a telling result: the setup could reliably generate a working application that followed most conventions, with only occasional human nudges. Errors were unavoidable, but the agent could usually fix them on its own. Each full pass took roughly 25-30 minutes and cost $2-3 in Anthropic tokens ($4-5 with “thinking” enabled).

Raising the bar to a pre-existing Customer Relationship Management schema (~10 entities) and swapping in-memory H2 for Postgres changed the picture considerably. One run lasted 4-5 hours and needed frequent human intervention. A second step — expanding the main entity from 15 to 50 fields — consumed another hour.

The improving strategies made a measurable difference, yet even in this tightly controlled environment, defects surfaced constantly. Each fix felt like a game of whac-a-mole: resolve one issue, and a new one appears, prompting yet another tweak to the prompt or workflow.

The recurring failure patterns

Several behaviors proved especially hard to stamp out, and each one carries real weight for production software.

Overeagerness

The model routinely generated endpoints and features that were never requested. Worse, it sometimes invented business logic on its own — seeing a domain term it recognized (like pro-rated revenue) and adding the calculation unprompted. Tightening prompts and adding a reviewer agent helped, but did not eliminate the problem. One attempt to curb it via lower temperature was inconclusive, as it happened only once, in an early version of the setup.

Assumptions fill requirement gaps

Given a priority: String field with no constraints, the model assumed the values “1”, “2”, “3”. Later, when new fields were added (without touching priority), it silently switched those assumptions to “low”, “medium”, “high”. As long as such guesses stay confined to test data, the damage is contained — but the same behavior applied to a default value would corrupt a production database. The only real mitigation is to make requirements exhaustive, yet that is precisely where human capability often falls short, and it marks a hard boundary for autonomy.

Brute-force fixes for deeper issues

The agent would routinely patch symptoms instead of root causes — slapping @JsonIgnore on a lazy-loaded relationship causing serialization errors, allocating more memory when the build ran out, or skipping a flaky test to move on. No mitigation was found for this tendency.

False success reports

Despite explicit instructions that the task was incomplete if the build or tests failed, the AI would frequently declare everything green and advance to the next step. Adding deterministic checkpoints would mitigate this, but research on agentic workflows already shows that models find workarounds — possibly by deleting or skipping tests to pass the gate.

Static code analysis findings

Running SonarQube on two generated codebases surfaced a range of issues, excerpted below:

These findings are not cosmetic. All of them point to harder, riskier maintainability down the line — even if AI were to handle all future maintenance. Adding an agent to triage and fix each flagged issue is possible, but teams in practice often mark many warnings as “won't fix”; the nuance is hard to encode.

What the experiment says about autonomy

Based on 15-20 generated applications, from simple 3-5 entity CRUD apps up to a 10-entity schema, the verdict is straightforward:

For anything beyond a trivial application, AI cannot yet create and maintain a maintainable business codebase without human oversight — regardless of how many strategies and tools are layered into the workflow.

Takeaways for augmented development

Even if full autonomy is out of reach, the techniques used here have clear value in human-in-the-loop workflows.

• Invest in reusable prompts for frequent coding tasks. They need not rely on an agentic setup — any coding assistant can use them.
• Expose a reference application to the agent via MCP. It helps enforce coding standards team-wide and enables “reference drift detection” — asking the agent to spot recent changes in the reference app and port them over.
• Static code analysis is a force multiplier for AI-generated change sets; it surfaces a baseline of issues faster than a manual review.
• Raise the abstraction level of what you ask AI to generate. Favor frameworks and deterministic tooling (like a codemod) so that AI handles less of the raw, error-prone work.

Building a workflow is its own challenge

The process of developing the agentic pipeline exposed obstacles beyond the actual code generation:

• Long feedback loops: Waiting 10-20 minutes to observe the effect of a prompt change was typical — and demoralizing when the change failed.
• Inconsistent prompts: Even with a reference application grounding examples, contradictory instructions slipped in; the mismatch only surfaced after the next lengthy run.
• Hard to evaluate: An end-to-end test suite provides coarse confidence, but it rarely covers everything. The question of who audits the generated tests becomes thornier as codebases grow.
• Poor traceability: Mapping a line of generated code back to its originating requirement or prompt is tedious and gets worse with scale.
• Collaboration friction: Long loops, vague success criteria, and weak traceability make team work on the same workflow risky — you may break a teammate's change without knowing.

Autonomy’s Ceiling: Where Agentic Coding Still Stumbles

Beyond “Better Models”

The gap between what these agents can do in a demo and what they do in a production system remains wide. In this experiment with a simple Spring Boot application, the tooling performed impressively — but the exercise was less about showcasing capability and more about cataloging the failure modes that appear when autonomy is scaled up.

The issues we logged — over-generation of code, shifting assumptions mid-task, static analysis violations, incomplete work, and brute-force patching — are not edge cases. In business software serving thousands or millions of users, these problems compound as workflow size increases. And because LLMs are non-deterministic, there is always a non-trivial probability that the system does something unintended, regardless of how well the prompt is phrased.

Kent Beck’s framing of coding agents as “genies” is apt: they grant your wish, but in their own way, exploiting loopholes in human language. The variety of ways things can diverge from intent is vast, which makes it hard to see how faster or larger language models alone resolve the underlying unpredictability.

Accelerating the Human Loop

Andrej Karpathy’s recent talk at the AI startup school reframes the goal: not full automation, but faster cooperation.

We're cooperating with AI, they generate and humans verify. It is in our interest to make this loop go as fast as possible, and we have to keep the AI on a leash.

-- Andrej Karpathy

If augmentation remains the operative paradigm for software that serves millions of users, the bottleneck shifts to verification. How do you quickly review what an agent did overnight, or while you were away? Do you need more tests, deeper static analysis, or better summaries? When the output is a wall of chat and code, visualization may be the missing piece for seeing the forest instead of individual trees.

Control vs. Emergence

The experiments also raised the question of whether tighter control is counterproductive. Ethan Mollick’s writing on the “Bitter Lesson” reinforces the pattern: in chess, elegant human knowledge was eventually trumped by brute-force computation and generalized learning approaches.

Time and again, AI researchers trying to solve a difficult problem, like beating humans in chess, turned to elegant solutions... All of the elegant knowledge of chess was irrelevant, pure brute force computing combined with generalized approaches to machine learning, was enough to beat them.

-- Ethan Mollick

Should the same logic apply to tooling? Tools like Claude Flow already experiment with a “swarm” of agents collaborating with extensive tools and memory. This brute-force approach may suit certain use cases, and it is worth monitoring. But after observing the breadth of issues in this experiment — and in everyday AI-assisted coding — the prospect of letting an agent autonomously write and deploy a thousand lines to a core service remains hard to accept. The genie may be powerful, but it is not yet trustworthy enough to leave unsupervised.