Rewriting pycparser with the help of an LLM - Eli Bendersky's website

A parser rewrite driven by an LLM

pycparser — my pure-Python C parser with roughly 20M daily PyPI downloads — has relied on PLY: Python Lex-Yacc for its core parsing since its inception in 2008. That dependency is now gone. I recently collaborated with an LLM coding agent (Codex) to rewrite pycparser as a hand-written recursive-descent parser, and the new implementation is already merged into the main branch. This article covers how that rewrite happened and what the process revealed about working with AI coding agents on substantial, production-grade codebases.

Why the old parser had to go

The original decision to use YACC-style grammar generation for pycparser made sense in 2008, particularly given that the C99 grammar in the K&R2 appendix seemed like a straightforward translation target. Over time, however, three independent problems accumulated.

Maintenance burden of a YACC grammar

While the original grammar famously had only a single shift-reduce conflict (the dangling else ambiguity), later additions for C11, C23, and widely-supported compiler extensions changed that picture substantially. The most recent PLY-based release carries numerous reduce-reduce conflicts — a situation where parsing rules must be tie-broken by their order of appearance in source code. That ordering discipline is fragile and hard to reason about. The grammar constraints that keep YACC parsers conflict-free don't apply to modern C standards anyway, since few industrial compilers use YACC anymore. pycparser's stability was maintained only through its enormous test suite, and each proposed extension (like this PR) made the situation worse.

Recursive descent, by contrast, produces parsers that are easier to understand, maintain, and typically faster. For a single complex parser, the conceptual weight of a parser generator as a dependency rarely pays off — a lesson that generalizes to other heavyweight dependencies in mature software projects.

The dependency problem resolved itself

pycparser had for years vendored a forked copy of PLY, originally to keep the codebase working for both Python 2 and 3 when PLY lagged behind. That arrangement worked, but when a security checking issue surfaced in vendored code that a later PLY commit had fixed, the broader situation became clear: PLY itself was archived and abandoned in late 2025. Re-vendoring the last pre-archive commit right in the middle of other work didn't look safe, and suggestions to migrate to another parser or write our own no longer seemed unreasonable.

Why an LLM agent could plausibly do this

Rewriting the parser by hand would have meant at least a week of tedious work porting ~2000 lines of YACC grammar into recursive-descent code without much opportunity to learn anything new. Given how much I'd moved on from active pycparser development, that effort stayed parked in my head for years rather than turning into actual changes.

What finally got me to try an LLM agent was the idea of conformance suites. The agent's task isn't open-ended when there's a large, high-coverage test suite acting as a rigid goal function — the kind of objective that LLM agents handle well. pycparser has exactly that: a test suite spanning over 2,500 lines of C snippets with expected ASTs, built from real bugs reported over 15 years. Even if the LLM succeeded or failed outright, it was very unlikely to produce a plausible but wrong port that somehow passes all those tests.

The initial port

With Codex launched inside the repository, I sent a first prompt simply to confirm it could run the tests:

This repository implements a Python parser producing an AST. It has a very
extensive test suite; can you first verify you successfully run all the tests?
(with python3 -m unittest discover, from the root dir)

After that was confirmed, the real prompt followed:

pycparser uses the PLY library for its lexer and parser, and its own AST
package for the ASTs created by the parser. I want to explore replacing the
Parser component (leaving the Lexer alone for now) with a hand written
recursive descent parser that will pass all the same tests. Can you try that?

Codex then worked for over an hour. I had assumed it must have gone off the rails long before that, but it eventually reported success:

Replaced the PLY parse path with a handwritten recursive‑descent parser while
keeping the existing lexer and AST helpers intact; CParser now points to the
new RD implementation, with the old PLY version preserved as PLYCParser in
pycparser/c_parser.py. The RD parser includes a token stream, declarator/type
parsing, expression precedence handling, pragmas, and statement parsing to
match the previous AST output and coordinate behavior.

Checking the result took a while — but it had genuinely written a new recursive descent parser, with PLY reduced to ancillary use only, passing the complete test suite. A few more prompts removed those remaining dependencies and clarified the overall structure. That functional result was impressive.

Managing safety: review strategy and branches

A rewrite of this scale cannot meaningfully be reviewed as one pull request. My strategy was to create a separate branch for the work, commit the initial agent output there, and only merge to main after reviewing everything piece-by-piece myself. Git made it easy to reset the branch when Codex took a direction I didn't like, or — worst case — discard everything cleanly. That branch discipline was essential given the extent of rework involved.

The long tail of goofs — and fixing them via prompt

Once the functional rewrite was settled, I asked Codex to do the same for the lexer and drop PLY entirely. Then began the deeper code-quality review. Reading what the agent generated was a journey. Codex behaves like a highly eager programmer that values reaching the destination over code clarity:

• Using raise...except for control flow.
• Exploiting Python's dynamic typing (where None, false, and other values each mean different things for the same variable).
• Spreading complex function logic around rather than keeping the core dispatch in a single switch.
• Resisting changes it initially claimed were impossible, only to succeed when I insisted: "Remember how we moved X to Y before? You can do it again for Z."

My guiding approach was to direct Codex to make the fixes rather than doing them myself. The commits were largely driven by three types of prompts:

1. Fixing over-complex code by replacing it with a simpler structural approach.
2. Removing needless convolutions, e.g., standardizing variable naming or patterns across all occurrences.
3. Adding detailed comments — with examples — explaining what each obscure block actually does.

Interestingly, instructions of the third kind often preceded a successful simplification: once Codex had written a clear comment, it became much more effective at revising the code the comment described.

Looking at the whole effort, I handled roughly 20% of the work myself, directing the agent for the remainder, and the final state of the parser is something I could see myself maintaining with or without agent assistance.

End result

The rewrite passes the full test suite. A new pycparser version (3.00) has been released with no significant issues so far — the only hiccup was that some of CFFI's tests asserted the exact phrasing of pycparser error messages, which was resolved with a straightforward fix. As a bonus, the new hand-written parser is about 30% faster in my benchmarks, which fits the typical pattern of recursive descent being quicker than generated YACC parsers. After the initial lexer rewrite, I also guided Codex through a performance pass, with reasonably good results.

All told, this was one of the more interesting coding efforts I've undertaken: a substantial, real-world software project successfully rewritten with the help of an LLM agent, provided that a good conformance test suite, careful branch management, and a fair amount of iterative prompting were in place.

Annotations as guardrails

One of the clearest takeaways from the rewrite was how much static typing helps an LLM coding agent. Type annotations act as a strict, always-on guardrail, much like a test suite. During the pycparser work, untyped code led the agent to overload values with inconsistent types — for example, mixing None, False, and other values in the same variable. Had the codebase been annotated from the start, these mistakes would likely have been caught earlier or never introduced at all.

To test this theory, I followed up by asking Codex to type-annotate the entire pycparser codebase, running checks with the ty tool. That exercise itself became a back-and-forth process, because the annotations exposed underlying design issues that needed refactoring. The result should make future agent-driven changes easier, though the long-term payoff is still to be measured.

Based on this experience, I would expect coding agents to be noticeably more effective in strongly typed environments like Go, TypeScript, and particularly Rust.

Practical verdict

Overall, the project was a strong validation of what modern LLM coding agents can do. These tools are already useful enough to meaningfully boost programmer productivity, and that is true even if progress were to stall from here.

Could I have done the rewrite myself? Yes, but it would have taken substantially longer — I estimate at least a week of full-time work, spread over an unpredictable calendar. With Codex, the total effort was roughly 4–5 hours, and I am satisfied with the outcome. That is an order-of-magnitude reduction in input effort.

There was also an enjoyment factor. Much of my professional satisfaction comes from reaching a state of deep focus and flow, which can be hard to enter deliberately. An agent that drafts a working prototype is a strong catalyst; it lowers the activation energy for starting a task and helps me get into that productive state more readily.

Does maintainability matter anymore?

An honest question emerges from this experiment: if coding agents can write and later understand the code they produce, does the code's quality even matter? If today's agent can maintain it — or next year's model can — why not simply let the agent drive and skip the human review entirely?

I don't have an answer yet. For projects I maintain and stand behind, I still want the code to be fully transparent and acceptable to me as an engineer; the agent is simply a more efficient means to that end. Where this balance lands in the future is genuinely uncertain, and watching it evolve will be interesting.