Amazon Tokenmaxxing: Why AI Usage Is a Bad Metric
Amazon warns that token counts and AI adoption can mislead engineering teams. Learn what its memo says and how to measure AI's real impact on delivery.
Amazon's AI Metrics Problem Is Bigger Than Tokenmaxxing
An engineer can burn through lots of AI tokens, generate more code, and rack up tool-assisted changes - without actually shipping more value. That gap is what sits behind Amazon's warning against tokenmaxxing: using AI as much as possible simply because usage has started to look like performance.
In a memo to software development engineers, Amazon argues that token consumption, AI tool adoption, and counts of code changes are shaky measures when they're treated like targets. The company isn't telling engineers to stop using AI, or managers to stop tracking anything. The point is a distinction many organizations miss as they roll out AI: a metric can help you understand work, but it shouldn't pretend to be the definition of good work.
And that distinction gets easy to lose once leaders need to prove an AI investment is "working." The result is often more activity, higher costs, and code that still needs plenty of review and maintenance. This article breaks down what Amazon's memo is saying, why usage metrics can push teams in the wrong direction, and how engineering leaders can judge AI by production outcomes instead.
What Amazon's tokenmaxxing memo actually says
Amazon's memo - signed by Terk Terkowitz, vice president and distinguished engineer for Worldwide Amazon Stores - focuses on how the company evaluates engineers' use of AI tools. CNBC reports that Amazon shared it with staff in September.
The memo's core message isn't "don't use AI." It's that raw measures of AI activity aren't automatically the same thing as useful engineering outcomes.
Amazon says it has used signals like token usage, AI tool adoption, code changes, and code reviews. Those can help you see whether teams are experimenting with AI or how work is flowing. But Amazon warns that these signals become unreliable when teams are asked to maximize them. If the goal becomes "consume more tokens," for instance, teams can end up putting AI into tasks where it doesn't really help - or at all.
The memo ties this risk to Amazon's leadership principles. It argues tokenmaxxing conflicts with frugality if teams spend more on tooling without producing useful results. Counting lines of code can also push teams toward bigger patches without necessarily improving anything - because more code isn't the same as better code, and small changes can sometimes remove real operational or customer friction.
If your work does not benefit from current AI tooling, that is a finding, not a failing.
That line matters. It works as a guardrail against blanket adoption mandates. Some tasks are a good fit for AI assistance. Others are too sensitive, too ambiguous, or too context-dependent for today's tools. If a task doesn't benefit, the right move is to figure out why - not force usage just to make a dashboard look healthier.
The memo doesn't throw the measures away. It says they correlate with outcomes that matter and should be used with care and judgment. Amazon also points to "frontier teams" - the teams that have adopted AI the most - as showing higher productivity and delivery. At the same time, it tells managers to look at the bottom 10% of teams by AI adoption and investigate why adoption is low. Leaders should also check whether the top 10% are delivering enough to justify their adoption.
That pairing is the key: adoption is a reason to investigate, not proof of performance. High usage could reflect a working workflow - but it still needs to show up in delivery. Low usage could reflect a gap, or it could reflect work that genuinely doesn't benefit from the available tools.
Why the memo is not an anti-AI policy
Amazon's stance goes beyond "use less AI." The memo expects AI to remain central to software engineering. It also expects leaders to use these tools on actual production work and to be able to point to where AI helped - and where it didn't.
That turns adoption into a management question, not a slogan. If a leader has used the tools in real tasks, they can talk concretely about where AI saves time, where it creates extra review burden, and where it's a poor fit. The goal is informed use - not a higher score on an activity chart.
Why AI usage metrics become bad targets
A metric starts to cause harm when people can improve the number without improving the underlying thing the metric is supposed to represent. This isn't unique to AI. It's a classic measurement problem in software teams: once a proxy becomes the target, behavior shifts toward the proxy.
Token consumption is a clear example. A higher count can mean engineers are using an assistant for useful work. It can also mean they're generating extra prompts, producing unnecessary output, or pushing a model to tackle a problem that didn't need model help. The metric doesn't tell you whether the result was accurate, merged cleanly, maintained well, or delivered real customer value.
Code volume has similar blind spots. More generated lines can increase the review surface and create future maintenance costs. In other cases, AI can help produce a small, focused change that fixes a production issue quickly. A line-count target can't capture that difference.
The incentive loop behind tokenmaxxing
Once a team is ranked by AI usage, that ranking can quietly become the work. People may drift toward tasks that are easy to run through an AI tool - even when those tasks aren't the most important ones. They may also split or repeat work in ways that inflate recorded activity. No bad intent is required for this to happen; it's a predictable response to incentives, especially when teams aren't sure how performance will be judged.
CNBC reports that Amazon previously shut down an internal leaderboard in May after employees were tokenmaxxing to climb the ranks. Business Insider also reported that Amazon tracked engineers' AI usage, token consumption, and lines of code, using leaderboards to rank some teams. The reporting described friction among employees. Taken together, these examples show the point: the behavioral impact of a metric matters as much as what the metric claims to measure.
The downstream cost is easy to miss too. More generated code often means more review, testing, and maintenance. More model calls can raise spend without improving delivery. If teams are rewarded for activity, leaders can end up funding a measurement system that creates work instead of removing it.
Why correlations still have a role
None of this means usage data is useless. Amazon says these metrics correlate with outcomes that matter, and correlation can help leaders spot patterns. If teams with higher adoption consistently deliver useful changes faster, that's worth investigating. If adoption varies wildly across similar teams, the differences may reveal gaps in tooling, training, workflow, or task mix.
The mistake is treating correlation as proof. A useful dashboard should prompt a question, not hand you an answer. Raw usage can be evidence to inspect, but it can't confirm that the work mattered on its own. The right interpretation depends on context: what the team builds, how risky changes are, and what "successful delivery" even looks like in that setting.
How should engineering leaders measure AI impact
There's no single number that captures AI's contribution to engineering. A practical evaluation blends activity signals with delivery, quality, cost, and human review. You don't need a perfect "AI score." You need to know whether a change in tools or practice improves a specific part of the work without creating larger costs elsewhere.
Start with a defined task or workflow. For example, a team might test whether an assistant helps engineers draft routine internal service changes, diagnose a class of test failures, or produce a first pass of documentation. Then establish a baseline before making the tool a standard part of the workflow. Without that comparison, teams can mistake workload shifts or staffing changes for an "AI effect."
Use an outcome-and-cost scorecard
A practical scorecard can use four buckets:
- Delivery: How long does it take for a useful change to move from work starting to production? How many changes actually make it to production successfully? Amazon vice president and distinguished engineer Marc Brooker has discussed measures like feature velocity, including launches and successful changes reaching production.
- Quality: What happens to defects, rollback rates, incidents, and the amount of rework after review? A faster first draft isn't a productivity gain if it just creates more downstream correction work.
- Cost: What does the workflow cost in model usage, engineering time, review time, and operational support? Token spend is one piece of the bill, not the whole thing.
- Adoption and experience: Are engineers using the tool on relevant work, and do they feel it helps? These signals can explain results, but they shouldn't replace them.
Use these measures together, not as a league table. A team working on a high-risk system may deliver fewer changes than a team shipping low-risk features - and that can be totally reasonable. Comparing raw throughput without accounting for the work can reward speed at the expense of reliability.
Run a small, credible comparison
Pick a workflow with enough repeated work to observe patterns, but don't assume every engineer or task is interchangeable. Compare similar work before and after AI is introduced, or run a limited pilot across comparable groups if the work allows it. Track task type, review effort, quality outcomes, and cycle time.
This isn't about claiming "laboratory-grade causality" from an ordinary team experiment. It's about reducing obvious confusion. If the work mix changes, a new dependency appears, or a release deadline shifts, those can explain the result as easily as AI can.
Then ask whether the benefit lasts beyond the initial learning period. Early adoption can add overhead as people learn the tool. Later, the workflow may stabilize and get more efficient. The reverse can also happen - quick early wins may fade if generated code later turns into maintenance and review drag.
Brooker's warning, as CNBC reports it, is basically the economic test: teams shouldn't spend tokens on problems that don't matter. An AI workflow works when it improves an outcome the business or users care about, at a cost and risk the organization can justify. Usage is something to inspect - not the finish line.
What the guidance means for engineers and the market
For engineers, Amazon's memo offers a solid way to push back on performative adoption. If AI doesn't help with a particular task, document why and keep meeting the task's requirements. Sometimes the work depends on domain context the tool doesn't have. Sometimes the assurance level needed makes generated output uneconomical. Sometimes the task is simply faster to complete without help.
The memo also suggests engineers should be ready to explain their AI usage in terms of the work itself. Don't just say "we used a tool." Describe what it produced, what needed correction, and what changed in the final result. That gives managers information they can act on - and it creates a clearer record of where the tool genuinely helps.
For managers, the lesson is to avoid turning adoption dashboards into performance rankings. Amazon's instruction to study both the bottom and top 10% is stronger than assuming low adoption means resistance or high adoption means success. Lower-adoption teams may have real constraints. Higher-adoption teams still have to show better delivery - not just more activity.
AI budgets are becoming an operating question
Amazon's memo lands as companies reconsider how they pay for AI use. Microsoft set division-level AI budgets in August. Uber's technology chief said the tokenmaxxing era was ending. Accenture reportedly capped staff Claude spending at $29 a month. Details differ, but the management challenge is the same: AI costs need to tie back to useful work.
A spending cap can limit exposure, but it doesn't answer whether the tool is actually valuable. A generous budget might encourage productive experimentation - or it might hide waste. The right policy depends on cost structure, how sensitive the work is, and whether the organization can measure results. Treating a fixed allowance as proof of value is just as misleading as treating a big token number as proof of productivity.
The trade-off is discipline, not adoption versus restraint
Two easy mistakes show up immediately. One is mandating AI use everywhere and calling compliance "innovation." The other is dismissing AI because early measurements look noisy. Amazon's guidance sits between these extremes: keep experimenting, look for where the tools improve delivery, and don't confuse usage with impact.
This also puts responsibility on leadership. If managers are expected to judge whether AI helps, they need first-hand experience with production work. Otherwise, teams get targets based on assumptions that haven't been tested against reality - code review load, security requirements, reliability standards, customer impact, and the time required to maintain what gets shipped.
For the market, the shift is from counting access and activity to asking whether AI changes the economics of delivery. That's harder than it sounds, and it tends to produce messier dashboards. But it's the question that matters when companies decide which tools to renew, which workflows to redesign, and where human review will always be essential.
Conclusion
Amazon's warning isn't that AI metrics should disappear. It's that activity measures become dangerous when people treat them like outcomes. Token counts, adoption rates, and code volume can show where AI is being used - but they can't tell you, on their own, whether the work got better.
For engineering leaders, the practical test is to connect AI use to delivery, quality, and total cost - and then dig into cases where those don't line up. For engineers, it's to use the tools where they genuinely help and explain where they don't. If you're setting AI goals for a team, start with the work you want to improve, not the usage number you want to increase.