Cutting our inference bill by 68% by being obsessive about optimization

The review's cost growth was not driven by a single failing component. Between March and August, the average price of a review climbed from $1.22 to $2.32, with intermediate values of $1.44 in June and $1.74 in July. Part of that reflects a shift in review mix: the most expensive tier accounted for 59% of reviews in March and 65% in August, which the team attributes to better models producing larger, more complex changes. Most of the increase, however, came from model changes rather than volume.

Moving every reviewer onto Anthropic models at the start of August raised cost per job from $1.38 to $2.03, a choice made because those models outperformed alternatives on this workload during evals. The internal standards reviewer, Codex, landed on Claude Opus 4.7 for a combination of performance and reliability, which took it from roughly $0.25 per review to $0.88 — more expensive than the coordinator agent itself. Meanwhile the cache hit rate improved from 85.7% in April to 97% by September, but tokens per review more than tripled over the same span, from 0.91 million to 2.99 million. A high hit rate on three times the tokens still costs more, even at a low cache read price.

Platform changes and a new runtime

The reviewer now runs on OpenCode v2, a ground-up rewrite of v1. Before migrating, every review in every canary repo ran on both versions in comparative shadow mode; the two agreed on the approval decision in 98% of tested merge requests, so the team moved everyone over in one go in early October and deleted the v1 code. On two days with identical models, full-tier reviews on v2 finish about 13% faster. Other execution approaches are under experimentation, including running the internal reviewer solely on Durable Objects using pi-durable.

Reading the token ledger

Just before optimization work began, spend broke down as roughly 23% output tokens, 43% cache writes and 34% cache reads — meaning most of the money went to context the model had already seen or was about to see again. A single session explains why: the code quality reviewer averaged 16.6 LLM calls per review, writing 72k tokens to cache and reading back 1.08m, since each call resends the whole conversation as context. Session cost grows with both the call count and the context size, and both had crept upward over the preceding months.

Anthropic's prompt cache stores already-computed prefixes — portions of the conversation at the start — so a later request can reuse everything up to the point where it diverges. Prefixes are bounded by cache breakpoints, writing costs 1.25× normal input price, reading costs 0.1×, and an entry expires after five minutes unless something reads it. The harness tools, system prompt and early messages rarely change in a reviewer session, making them strong caching candidates. That left two levers: make the first request of each session read from cache rather than write, and reduce calls per session.

One prefix for every repository

From September 14, the reviewer logged token usage on each session's first LLM call along with a hash of the first system block as sent to the provider. The data showed the first request found a cached prefix only 14% to 26% of the time, despite every session opening with the same tool definitions and agent prompt — between 11k and 21k tokens identical across all repositories.

The cause was OpenCode v1's system prompt construction: it built a single block holding the agent prompt followed by an environment section with working directory and date, and placed the sole cache breakpoint at the end of that block. The shared portion was never stored independently, so no repository could reuse another's cached prefix. On September 16 the team added a marker to the end of every agent prompt and a plugin hook that splits the block in two at that marker. The first half now ends on bytes common to every job and carries its own breakpoint. Keeping it that way required removing repository-specific content from in front of the marker, so the available-skills list is disabled and tool descriptions cannot vary between jobs.

Also on September 16, the Codex reviewer's reference index — a table of internal standards, a skill file and a list of reference files — moved into its prompt. Previously each session spent its first three turns reading those files via tool calls. Calls per run fell from 15.9 to 9.6 and cost per run from $1.12 to $0.72. Combined, these changes took cost per review from $2.47 to $2.02, with the effect visible as soon as the release shipped.

Consolidating calls, consolidating prefixes

Releases on September 21 and 23 were individually unremarkable but collectively removed about $0.20 per review.

The most involved change shares a single prefix across three reviewers: code quality, security and performance now send identical first requests containing the same tools, a shared rules file, and the merge request with all patches included. Each reviewer's own instructions follow, and the team ensures one reviewer writes the cache while the other two read from it.

Because a cache entry cannot be read until the response that wrote it has begun streaming, and all three reviewers start simultaneously, the second and third wait up to 20 seconds for the first to respond before sending their own first request. The added latency buys roughly nine cents per reviewer.

Models and effort levels

Splitting the period at each release and configuration change shows code changes account for $0.65 of the $1.67 total drop, while model and effort changes account for $1.06. Each step compares weekday hours before a change with hours after, so this is not a statistically sound controlled experiment.

The coordinator had run three model and effort combinations since August. On the same model, full-tier jobs at medium effort finished 20% sooner, so the other two were dropped. A later randomized split found Opus 5.5 at medium showed no measurable change in review outcomes or critical findings compared with Opus 5, prompting a switch; Opus 5.5 at high effort cost more without a meaningful increase in findings. Opus 5.5 is cheaper per token than Opus 5, but price accounts for only about $0.20 of the drop — the rest comes from sending and generating fewer tokens. Since early September, cache reads per review are down 82%, cache writes down 54% and output down 67%.

On October 3, every Sonnet reviewer was split 50/50 between Sonnet 5 and Sonnet 5.5. After two and a half days, Sonnet 5.5 was 62% cheaper per run for code quality, 58% for performance and 47% for security. Both models cost the same per token, so the entire saving comes from fewer tokens. On full-tier reviews the 5.5 reviewers posted about as many critical and warning findings as Sonnet 5, and on October 5 all were moved to 5.5.

At medium effort on Sonnet 5.5, the code quality reviewer makes 1.7 LLM calls per run instead of 6.2, and more than half its runs finish in a single call — opening no other file in the repository. At high effort it makes 3.4 calls per run. Since October 5 the code quality reviewer has been split 50/50 between medium and high effort on 5.5; after one day, high effort cost 61% more per run and posted 58% more critical and warning findings. That works out to about two cents per review, and since those findings proved worth having, code quality is moving to high effort. The cost figures in this post assume that move.

Did the savings come at the cost of detection?

A cheaper review is only useful if it still catches problems, and the initial finding counts suggest it might not.

Code quality on Sonnet 5.5 at high effort now reports 40% fewer critical and warning findings than in late August. Security and performance, which remain at medium effort, report 56% and 71% fewer. A drop in severe findings alone cannot distinguish between problems being missed and noise being filtered out, so we examined it from two directions.

For the first, we checked whether authors act on what the reviewer flags. We compared the reviewed commit with the author's next push across samples of critical and warning findings, counting a finding as acted on when the lines it pointed at had changed. Lines 30 to 40 away from each finding changed about 15% of the time in every period, giving a baseline for coincidental edits.

Before the changes, authors acted on 34% of the code quality reviewer's serious findings; now the figure is 59%. Fewer things get flagged, but those flagged are far more likely to lead to a change, so findings that led to a change per 1,000 reviews is roughly unchanged: 132 versus 145 before, within the margin of error. Across all four reviewers, serious findings that led to a change rose from 229 to 321 per 1,000 reviews.

Second, two models from different providers judged a blinded sample of 657 findings from the three Sonnet reviewers, each with the diff it referred to and no indication of which period it came from. Disagreements went to a third model. Findings were labelled important, minor, duplicate, wrong or unverifiable, and severity was marked fair, too high or too low.

Most of what vanished was minor findings, along with some duplicates and findings the judges could not verify from the diff. Important findings per review moved from 0.28 to 0.30, with overlapping intervals. Severity shifted far more: the judges previously rated 0.46 critical or warning findings per review as more severe than deserved, and that is now 0.14 — a 70% drop and the clearest result in the research. Much of the decline in critical findings comes from the reviewer no longer labelling minor issues as critical.

Where each reviewer stands

The answer differs by reviewer. For code quality at high effort, mostly noise and inflated severity disappeared, and it still finds about as many real problems as before. Medium effort, though, lost real findings: in the randomized split, code quality at medium led to roughly a quarter as many acted-on findings as at high, which is why it moves to high. Performance also lost real findings — the judges found about 64% fewer important performance issues per review, in practice about one fewer important finding every 24 full-tier reviews. Security fell by about a quarter, which is not statistically significant. Both remain at medium effort, and testing them at high comes next.

Neither method is perfect. An author changing flagged lines shows they did something, not that the finding was right. The judges are models too, and each group holds only 35 to 66 findings. The "now" window is also just one day long. Still, the two methods agree, and neither suggests code quality is finding fewer real problems.

Reliability held up as well: reviews ending without a verdict went from 0.79% to 0.27%, and security reviewer retries from 3.5% to 0.04%.

Cost and latency today

An average review now costs $0.77 rather than $2.43, and the median finishes in 1m 32s instead of 4m 9s. The largest single saving came from checking what actually reached the provider on each session's first request, then ordering those messages so they stay as consistent as possible across commits, merge requests and repositories. The rest came from sustained experimentation with model and effort combinations, deliberately setting aside any preconceived preference for particular models.

Cheaper reviews did not automatically mean worse quality — but they did not automatically mean acceptable quality either. Code quality at high effort finds about as many real problems as before, with less noise and far fewer inflated severities. Performance lost real findings at medium effort, and security may have, so testing both at high is the next step. For anyone running agents at volume, the first two things worth checking are whether each session's first request actually reads from the cache, and whether your reviewers still find real problems at the effort level you are paying for.