Astra changes the test-technique picture
In the August run of the agentic test/verification eval, GPT-5.6 Sol produced what amounted to caricatures of whatever technique it was told to use — TDD, QuickCheck, Lean 4 — while the condition with no prescribed method (Default) generated plenty of worthless unit tests yet still ranked among the best outcomes, because naming a technique mostly hurt. That eval handed agents the Zstd RFC with an addendum describing how to test for every non-Default condition. Re-running it against GPT-6 Astra reveals two shifts: capability now exceeds the eval, and the techniques themselves are being applied more competently.
The Zstd task is effectively saturated. Under 5.6 Sol, medium-effort runs averaged $3.50 and xhigh $5.50, reaching full holdout-test correctness roughly 63% and 77% of the time. Astra hits full correctness over 99% of the time at medium effort for $1, and nearly every condition lands near 100% regardless of technique. The improvement in technique use is visible even where it can't be measured: Sol's fuzzing generated random inputs that mostly re-hit the same paths, whereas Astra fuzzes in a structured way that exercises real code paths and surfaces non-trivial bugs — though with correctness pinned at 100%, the Zstd benchmark cannot show whether that mattered.
IMAP and Pandoc: testing on unsaturated tasks
The IMAP RFC had defeated 5.6 Sol outright; with Astra it yields a spread of results. Across 80 runs per condition, Default again looks strong: several conditions edge it out on correctness but at higher cost. Because the IMAP RFC is unsaturated but still not hard enough, the harder Pandoc eval from the programming-language comparison was brought in — a deliberate choice, since models behave differently on larger tasks, as seen when caveman-mode gains shrank and when C and C++ Zstd implementations were at least somewhat memory-safe while their Pandoc counterparts were plagued by safety issues.
Pandoc is hard enough that no condition reached 100% correctness even after multiple turns (about 14 on average), so the metric becomes the fraction of tests passed rather than the fraction of runs achieving a perfect score. Each condition ran only 25 times, at a cost of roughly $90k at MSRP.
Earlier language and technique evals made a recurring point: failures were often idiosyncratic artifacts of agent behavior — Clojure-using agents writing incorrect byte conversion code by default, Verus-using agents writing vacuous proofs. Those are distinct from the human-sounding rationales people give for preferring a language or technique, which tend to sound appealing but look unrelated to what agents actually do. One recent example: the claim that Rust suits agents because its error messages guide them to correct solutions, which doesn't hold up in any problem examined at affordable sample sizes.
Reading the Pandoc results
The bootstrap intervals understate the true uncertainty, and $90k of evals is not enough to explain why nearly anything happened. We can see that some conditions beat others, not what it means. Two observations still line up with expectations: techniques that should generalize better should score higher on holdout than visible tests (above and to the left of the Default line), and TDD should generalize worse (below and to the right) — and TDD does underperform. But the built-in condition would then also be expected to show a low holdout-to-visible ratio against Default, and it doesn't; resolving that inconsistency would take far more runs. The cost-versus-score curves are fitted to the shape of all multi-turn trajectories, fit to Default, so points below/right are worse and above/left better.
Under 5.6 Sol the verdict was clean: almost no condition beat Default, similar prompts failed to beat it on other evals, and the non-Default conditions produced mostly ridiculous tests that didn't capture the technique's value. Here many techniques do beat Default, so even with wide error bars it is plausible that one or more genuinely do — we just can't say which — and some conditions likely outperform others. A uniformly bad field is much easier to interpret.
Open questions remain. Do rankings persist with more turns toward near-complete Pandoc correctness? On average conditions keep roughly their rank order with additional turns, but that's untested here. How much comes from 25 runs per condition, given heavy path dependence when looping over generated code? How much from minute prompt variations — sentence structure, capitalization — which have produced very different outcomes in quick experiments? And how representative is one Pandoc-sized problem of a hundred? Answering these properly means running at least 100× this matrix; Steve Yegge's reported ~20 Claude subscriptions at ~$120k/mo of MSRP tokens won't get there, though 2×–3× is feasible with limited expansion and extrapolation. A grad student might find it worthwhile next to some of the widely cited programming-productivity studies; in industry, it's hard to picture anyone but an AI lab doing it.
What stood out, and the caveats attached
- Differential testing pushed agents to examine tests and boundary conditions closely enough to implement both sides of a boundary correctly — something the hastily written 5.6 Sol Zstd skill failed to elicit.
- TDD underperformed again. Yet other conditions that also produce many small tests did not underperform, weakening the tidy explanation that lots of small tests are the problem.
- ACL2 did well, as it did in the 5.6 Sol Zstd eval. The author rates ACL2 as underrated versus other verification tools and remains only low-confidence about that; with all the effort going into Lean, Lean should eventually surpass it, but today ACL2 looks like a reasonable choice, and comparisons from Lean 4 users would be welcome.
- Property-based testing finally produced tests that behave as PBT is supposed to — including Proptest, which performed well again — though without the named PBT condition.
Confidence that any of this generalizes is low. The guess is that no major work on PBT or fuzzing landed between 5.6 Sol and 6 Astra, so the gains reflect better generalization and higher capability — which, if true, suggests models will keep improving at testing without targeted effort.
ok it's official...[ellipses from original quote] AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
That viral claim rests on DeepSWE, a benchmark this author already considers unrepresentative, run with Sonnet 5.5 High — so it covered only problems far easier than those above, and technique effectiveness is known not to generalize across problem size or difficulty. What DeepSWE actually showed: asking for no tests gave similar correctness and a small 9% average cost saving. Here and in earlier evals, correctness differences appear alongside cost differences larger than 9%. DeepSWE's grader is also broken for testing: among other issues, an agent can add a test name colliding with a grader secret test name, causing failure, so writing tests is penalized in a way real usage isn't. Overstated "empirically proven" claims do travel — the tweet had 2.8k likes, 2.8k saves and 395 retweets within a month — but they're a poor way to learn how to drive agents. As Nolan Lawson put it, the point is that decisions can still be made on evidence rather than vibes; nobody, AI lab or otherwise, could call X or Y empirically proven helpful or unhelpful, but toy evals yield some information.
For practice: Proptest is the pick for property-based testing in Rust; for any problem where Lean would be used, trying ACL2 as well to compare results seems worthwhile. No per-condition accounting like the 5.6 Sol Zstd writeup accompanies these results — that post was too long to reconstruct when every eval writeup is meant to take 30 minutes and results are outpacing the time to publish them.
Appendix: skill portability and field reports
Why don't AI labs publish more comparisons of effective techniques? They can afford comprehensive evals, the information is scarce, better usage should drive more usage, and techniques that work on OpenAI's models differ from those that work on Anthropic's — publishing this could steer people toward modes of interaction that favor the publishing lab. The counterargument: revenue growth is so fast that other work pays better, and prompt differences don't matter. The first is plausible; the second can't be, given how repeatedly reasonable people without obviously silly habits get wildly different results. If the effects are real, their impact exceeds a skimming of the graph because more effort, turns and dollars yield diminishing correctness returns — visible in the shallow slope and curvature of the Default fit line. Still, the differences between how people prompt in practice dwarf what a few words about test technique produce, which oddly makes those prompt differences hard and expensive to eval.
David R. MacIver observed that ChatGPT is far less inclined toward PBT than Claude, which writes property-based tests eagerly and seems to benefit during development; whether Claude is actually better at PBT is unverified — only more proactive — and it may be prompt sensitivity or model specificity. Testing hasn't been done at comparable breadth with Claude, and Hegel (which MacIver develops, largely with Claude) might simply work better there than with Codex. Notably, both Hegel skills — the original used in the 5.6 Sol Zstd eval and the update produced afterward — fared worse than Hegel with no skill: similar correctness at much higher cost, with the updated skill costing more, so a skill improved for some cases got worse for others. Similarly, the quickly written Skill condition that outperformed under 5.6 Sol underperformed with 6 Astra on Pandoc; someone experienced with Astra on larger problems could probably write a better one, but that hasn't been done here. And long familiarity with Codex gives little sense of how effective Claude would be in practice.
Roman Leventov reported a case study: rewriting a TypeScript statechart-like support-chat system (deterministic transitions, an LLM classifying intents, external events), he ran Sol xhigh for five days and then mostly Astra medium or high for four to five more, with the agent generating event and effect traces for fuzzing. For the first four days the generated tests found no de novo defects that agent reviewers hadn't already caught, using a review approach of roughly 15 angles/scopes with subagents prompted verbatim from templates until defects ran out. Agent reviewers found about 210 bugs across the rewrite; generated tests eventually produced roughly 20 de novo defects, for about 230 total. The production change was about 30k lines, while the generated harness and supporting code reached about 300k lines — likely very ineffective, though not investigated. After deploy, 4–5 "stupid" bugs remained; he still counts it a good result for a rewrite of this complexity, but views it overall as negative: generated tests caught just 10% of defects at gargantuan code and token cost. More such case studies are wanted.



