What We're Learning About AI-Assisted Software Delivery

Thoughtworks has been coordinating research into how generative AI and large language models will affect software delivery practices. This series of memos captures what we're learning as we experiment with these tools in real engineering contexts — from agentic workflows and local models to the economics of refactoring with token costs in mind.

Coding Agents and Developer Loops

Kief Morris has been thinking about how AI agents fit into software engineering workflow loops. His memo focuses on the goal of turning ideas into outcomes by building and managing the working loop rather than merely automating isolated steps. In a separate experiment, a team working with fully agentic engineering practices unexpectedly prompted their agents to create a blackboard coordination system inside the git repository — an accidental emergent pattern worth examining.

In "Autonomous Coding Agents: A Codex Example", we look at how OpenAI's Codex behaves under the hood and which agent category it falls into. Birgitta Böckeler puts the move toward agents in context for developers:

LLMs are NOT compilers, interpreters, transpilers or assemblers of natural language, they are inferrers.

That framing carries through several memos about the importance of developer skills. A list of AI "missteps" illustrates how engineering judgment still matters in agent-assisted coding, and the risk assessments developers make while reviewing AI-generated output are the subject of an entire memo about probability, impact and detectability of errors.

Measuring Quality and Value

A handful of contributions examine quality and economics of AI-assisted coding more concretely:

  • A small, informal evaluation of whether prompting a coding agent to follow TDD inside its agentic loop actually improves resulting code and tests compared to no TDD instructions.
  • An experiment on whether decomposing a large function helps reduce token costs, suggesting it may now be possible to measure the economic benefit of refactoring.
  • An assessment of internal code quality by adding a feature with a coding agent to CCMenu and seeing what happens to the code.
  • A look at the common metric of AI code acceptance — Matteo Vaccari argues it has a big hole, because LLMs can be helpful even when you throw away their code.

On TDD specifically, Paul Sobocinski asked GitHub Copilot for help with test-driven development. His finding: Copilot can suggest useful ideas for the red and green steps, but it isn't very helpful for the all-important refactoring step.

Context, Harness and Specifications

Context engineering has become a huge part of the developer experience with modern LLM tools. One memo explains the current state of context configuration features, using Claude Code as an example. OpenAI's recent write-up on "Harness Engineering" is another valuable framing — covering context engineering, architectural constraints, and garbage collection of the codebase as key activities in AI-enabled software development.

Spec-driven development gets its own treatment through three tools: Amazon's Kiro, GitHub's spec-kit and the Tessl Framework. Related work looks at anchoring AI agents to a reference application to make template updates easier to feed back into already-instantiated services.

Local Models and Specialized Tooling

Two memos dig into the viability of local LLMs for programming: one lists the factors that influence how viable they are, and a companion piece compares local models with two standard tasks while trying the best LLM in day-to-day use.

Several engineering journeys demonstrate what's possible with real code:

  • Unmesh Joshi describes iterating with an LLM to handle the boilerplate of an extension to PlantUML, showing how even an experienced developer can get past plumbing with AI assistance.
  • Another memo shows how using an LLM helped him learn an unfamiliar toolset while building the TMT Mirror Visualization.
  • A developer-LLM dialogue on robust Java ByteBuffer code demonstrates how expert guidance is critical — AI editors generate code with remarkable speed, but more is needed for robust systems.

Scope, Onboarding and Newer Capabilities

On multi-file editing, the release of that capability in GitHub Copilot expands AI assistance to larger changes spanning many files. Birgitta's test of it examines how useful it is and what it means for developer feedback loops. A different memo describes building an AI agent application to migrate a tech stack by changing the testing framework in a test — a side effect being a better understanding of how these agents work.

For teams facing existing, messy codebases, one memo reports on the parts of the onboarding journey where AI was genuinely useful (and where it fell short). Even without AI in the picture, service templates have always suffered from the difficulty of feeding template updates to instantiated services; anchoring agents to the template may change that. And a start-to-finish case study of understanding and solving an issue in a real-life codebase shows where today's tools show promise.

Broader Risks

Agentic coding assistants expand the attack surface of the software supply chain, and a memo by Jim Gumbley and Lilly Ryan explores the safeguards needed when using them. On adopting the tools at all, Birgitta ponders how LLM-assisted code generation differs from traditional code generators, and at which software abstraction levels this technology sits.

The Human Factor

A few contributions push back gently on the prevailing enthusiasm. Coding assistants do not replace pair programming, and in-line assistance can get in the way when it amplifies bad practices, hinders refactoring by getting stuck in the past, or fatigues developers who must constantly review suggestions. A very early experiment asking GitHub Copilot to write a median function produced three flawed options — evidence you still have to know what you're doing when asking LLMs to write code.

On reasoning models, the recent buzz around R1 suggests a step change for coding assistance, but the memo asks whether they deliver on the promise:

A lot of stake is put into how reasoning models are a step change in coding assistance, especially the recent release of R1 has created additional buzz around that. But I don't see it — yet?

You can work through all of the memos in the Exploring Gen AI series — the very first explains a useful mental model of the AI tool landscape and how the pieces fit together.