AI Inference Cost Optimization: A 68% Reduction
See how Cloudflare cut AI inference costs by 68% while speeding code reviews—with prompt caching, fewer calls, and model changes. Apply the same method.
How Cloudflare Cut AI Inference Costs by 68% Without Sacrificing Review Quality
An AI feature can feel cheap right up until people start relying on it. Cloudflare learned that the hard way with its AI code reviewer: the average review cost climbed from about $1.19 in April to roughly $2.46 by late August. The reviewer's architecture didn't change much - the bill did.
The team dug into where the tokens were going, then adjusted how the reviewer built requests, reused context, picked models, and handled repeated reviews. The outcome: an average cost of $0.77 per review - about 68% below the $2.43 average during the two most expensive weeks before the changes. Median review time also dropped from 4 minutes 9 seconds to 1 minute 32 seconds.
The lesson isn't just "use a cheaper model." Most of the savings came from cutting avoidable work and testing model choices against real review outcomes. Below is how the team approached it, what prompt caching changed, how they checked quality, and what product teams can borrow for their own AI workloads.
Why the inference bill grew
The spike didn't trace back to one big architectural mistake. Instead, several things piled up over time: reviews got more demanding, token use increased, and the team moved the reviewer to models that performed better on this workload - even though they cost more to run.
The average cost per review rose steadily: from $1.22 in March to $1.44 in June, $1.74 in July, and $2.32 in August. At the same time, a larger share of reviews landed in the most expensive tier, increasing from 59% to 65%. That helped, but it didn't explain most of the increase.
Better models can create larger workloads
As models got better, engineers used them for more complex code changes. Those changes were more likely to require a full review, with multiple specialist reviewers focused on code quality, security, performance, and internal standards.
Cloudflare also made a deliberate provider/model change. At the start of August, it moved the reviewer to Anthropic models after internal evaluations showed better performance. That shift raised the cost per job from about $1.38 to $2.03. Its internal standards reviewer, Codex, moved to Claude Opus 4.7, taking its cost per review from roughly $0.25 to $0.88.
Paying more for a model can make sense if it finds more important issues or reduces the amount of human review required. The problem was figuring out whether the extra spend actually bought enough value - and where the rest of the tokens were going.
Cached tokens still have a cost
By September, the reviewer used about 2.99 million tokens per review, more than three times the 0.91 million reported in April. Cache hit rate also rose - from 85.7% to 97%. Those numbers only seem contradictory if you assume a high hit rate makes token volume irrelevant.
It doesn't. Cached inputs are cheaper than resending the same content as fresh context, but they aren't free. Before optimization, output tokens accounted for about 23% of spend, cache writes 43%, and cache reads 34%. Most of the bill was still coming from input context the model had already seen - or was expected to see again.
On average, the code quality reviewer made 16.6 model calls per review. Across those calls, it wrote about 72,000 tokens to cache and read around 1.08 million. Each call carried forward the conversation so far. As the number of calls and context size grew, costs stacked up.
So the first operational question wasn't "Which model is cheapest?" It was: Are we sending the same useful context over and over, and can the system reuse it more effectively?
How prompt caching and code changes reduced waste
Prompt caching lets a provider reuse computation for a stable prefix at the beginning of a prompt. In Anthropic's setup, a cache breakpoint marks where the reusable prefix ends. Later requests can reuse that prefix only when the content matches up to the breakpoint; anything after the request changes won't be reusable as part of that prefix.
That means prompt ordering matters. In the Cloudflare setup described, writing a cache entry cost 1.25× the normal input price, while reading it cost 0.1×. Entries also expired after five minutes unless read. If a system keeps writing nearly identical context instead of actually sharing it, you can end up paying a premium for work you meant to avoid.
Make the shared prefix genuinely shared
Cloudflare measured the first model request in each review, including a hash of the system prompt as sent to the provider. The result was eye-opening: only 14% to 26% of those first requests actually found an existing cached prefix, even though each session started with roughly 11,000 to 21,000 tokens of common tool definitions and agent instructions.
The issue was how OpenCode v1 assembled the system prompt. It bundled the shared agent instructions together with changing environment details - like the working directory and date - into one block, with the cache breakpoint at the end. Since that whole block varied from job to job, a review in one repository couldn't reuse the shared beginning generated in another.
The team fixed that by adding a marker after the shared instructions and splitting the block with a plugin hook. They also removed repository-specific, changing material from before the breakpoint. After that, first-request cache hits rose to between 91% and 99.5%.
If you're using prompt caching, the practical takeaway is simple: cacheability is partly an architecture and prompt-ordering problem. A great cache hit rate on later requests doesn't prove the first request is well designed. Measure the actual request payloads and cache behavior, rather than assuming the framework handled the prompt structure for you.
Remove calls that do not add useful work
After improving caching, the team also cut down unnecessary model interactions. Some changes were small on their own, but together they removed about $0.20 from the cost of each review:
- Remove unavailable tools from sub-reviewers. Sub-reviewers could see a tool for posting the final comment, even though they weren't allowed to use it. In 57% to 76% of sessions, they tried anyway, hit an error, and spent another call recovering. Removing the tool also cut about 2,900 tokens from each sub-reviewer request.
- Read files in parallel. The code quality reviewer often read one file per turn. It was changed to read shared context and relevant patches together in its first response, up to 150 KB.
- Skip reviewers on repeat reviews. About two-thirds of jobs were a second or later review of the same merge request. When nothing relevant had changed, some specialist reviews were skipped.
- Stop after posting the result. The agent made an extra call to summarize a review after posting it. No one used that summary, so the call was removed.
- Cancel work that has been canceled. In a sample of 446 canceled jobs, 358 kept running until completion because the execution environment didn't notify the script. Jobs now check their status every 15 seconds and stop when canceled.
There was also a more involved optimization: three reviewers could share an identical initial request - tools, shared rules, and the merge request patches. One reviewer wrote the cache entry; the others read it before adding their reviewer-specific instructions. Since the cache couldn't be read until the writing response started streaming, the other reviewers waited up to 20 seconds. That added a small latency trade-off, but saved about nine cents per reviewer.
In total, code changes accounted for about $0.65 of the $1.67 reduction, from $2.44 down to $0.77. That helped a lot - but it wasn't the whole story.
How model and effort choices changed the economics
The remaining savings came from changing the models and the effort levels used for different reviewers. This wasn't a blanket "pick the cheapest model" move. The team compared alternatives and tracked cost, latency, and review outcomes.
One key finding: on the same model, full-tier reviews at medium effort finished about 20% sooner than the other tested model-and-effort combinations. That led the team to simplify the coordinator config by dropping slower alternatives. In a later randomized split, Opus 5.5 at medium effort didn't show measurable changes in review outcomes or critical findings compared with Opus 5, so the team switched. Higher effort on Opus 5.5 cost more without a meaningful increase in findings for that role.
The model's lower per-token price explained only about $0.20 of the savings. The bigger win came from using fewer tokens. Since the start of September, cache reads per review fell 82%, cache writes fell 54%, and output tokens fell 67%.
Compare models on the task, not on reputation
Cloudflare also ran a 50/50 test between Sonnet 5 and Sonnet 5.5 for the Sonnet reviewers. After two and a half days, the newer model was 62% cheaper per code quality run, 58% cheaper for performance, and 47% cheaper for security. The per-token price was the same, so the difference came from token usage rather than a cheaper rate. The reviewers produced about the same number of critical and warning findings in the initial comparison, and the team moved to the newer model.
This kind of result doesn't generalize automatically. A model that needs fewer calls for code review might behave differently for legal analysis, customer support, or data extraction. Model selection should be treated as an empirical product decision - not a permanent ranking of providers.
Effort settings need the same level of scrutiny. At medium effort, the code quality reviewer averaged 1.7 model calls per run, down from 6.2 previously; more than half of runs finished in a single call. At high effort, it made 3.4 calls. Fewer calls helped cost and speed, but it raised a real concern: was the model doing less work because it was more efficient, or because it was missing issues?
A split test found that high effort cost 61% more per run and produced 58% more critical and warning findings than medium effort. The team decided those extra findings were worth the added cost for code quality, so high effort was used there. That's the trade-off worth making explicit: spend more where the quality gain matters in practice, instead of applying one effort level to every agent by default.
Overall, the headline reduction was about 68%: an average of $2.43 during the two most expensive pre-optimization weeks versus $0.77 after the changes. The reported breakdown attributed $0.65 to code changes and $1.06 to model and effort changes. These numbers came from comparisons around releases and config changes - not a perfectly controlled experiment - so they're best read as directional.
How to check whether cheaper AI is still good AI
Cost per request is straightforward to measure. Quality isn't. If a cheaper model misses important problems, some of those costs show up later - on engineers, on customers, or during incident response. On the other hand, fewer findings don't automatically mean the system is worse; it might just be producing less noise.
Cloudflare assessed quality using two approaches. First, it checked whether authors acted on serious findings. For each sample, the team compared the reviewed change with the author's next push and treated a finding as "acted on" if the lines identified by the reviewer changed. Lines around 30 to 40 lines away changed about 15% of the time in each period, which they used as a rough baseline for coincidental edits.
For code quality, authors acted on 34% of serious findings before the changes and 59% after. The reviewer flagged fewer critical and warning issues, but the issues it did flag were more likely to lead to a code change. The number of acted-on findings per 1,000 reviews was similar: 132 after versus 145 before (within the margin of error). Across all four reviewers, serious findings that led to a change rose from 229 to 321 per 1,000 reviews.
Second, two models independently reviewed a blinded sample of 657 findings from the three Sonnet reviewers. They categorized findings as important, minor, duplicate, wrong, or unverifiable, and they also checked whether severity was fair. When there was disagreement, a third model made the call.
Overall, the results suggested much of the drop was minor, duplicate, or unverifiable findings. Important findings per review stayed similar: 0.28 before and 0.30 after, with overlapping intervals. The clearest change was severity inflation: findings judged to be overstated as critical or warning dropped from 0.46 per review to 0.14 - a 70% decrease.
The conclusion varied by reviewer. Code quality at high effort found about as many real problems as before, while producing less noise and fewer inflated severities. Medium effort did lose real findings in the randomized comparison, which supported moving code quality to high effort. Performance at medium effort showed about 64% fewer important findings per review; security fell by roughly a quarter, though that result wasn't statistically significant. Testing both at high effort remained the next step.
These measurements have limits. An author changing a line doesn't prove the reviewer was right, and model judges aren't ground truth. Some groups contained only 35 to 66 findings, and the "after" window was short. The practical point isn't that these methods settle quality conclusively; it's that cost optimization came with outcome checks, and the team kept paying more only where the evidence suggested it mattered.
Latency improved alongside cost. Median review time fell from 4 minutes 9 seconds to 1 minute 32 seconds, and the 99th percentile fell from 15 minutes 20 seconds to 4 minutes 41 seconds. Reviews taking longer than five minutes dropped from 37.9% to 0.7%. Fewer calls reduced both token spend and the time spent waiting for sequential model responses.
Conclusion: optimize the system, not just the model
Cloudflare's result came from treating inference spend like a systems problem. Prompt construction, cache boundaries, tool access, repeat-review logic, cancellation behavior, model choice, and effort settings all affected the final bill. Along the way, review speed improved too, and the quality checks showed where lower effort was acceptable - and where it wasn't.
If you run AI agents at scale, start with two questions: does the first request actually reuse the shared context you think it's reusing, and does the effort level you're paying for still find the problems that matter? Then measure calls, tokens, latency, and outcomes together. A cheaper run only counts as an optimization if the task still gets done well.
For the original implementation details and measurements, read Cloudflare's report on inference optimization.