Amazon AI Guidelines: Measure Outcomes, Not Tokens
Amazon's AI memo warns against tokenmaxxing. Learn why usage metrics mislead, how to measure real productivity, and when AI is worth using.
Amazon's AI Memo Offers a Better Way to Measure Engineering Work
When a company rolls out AI tools, usage stats are usually the first thing people track. Tokens consumed. Engineers adopting the assistant. Lines of code generated. Those numbers are easy to put on a dashboard - but they don't answer the only question that matters: did the work get better or faster in a way that actually helps customers or the business?
Amazon's guidance to software engineers draws that line clearly. In an internal memo circulated in September and later posted on the company's website, Amazon warned employees against "tokenmaxxing" - using AI as much as possible without enough regard for the quality of what gets produced. The memo also argued that token use, adoption rates, and code changes are weak measures if you treat them as standalone proof.
This isn't just advice for software teams. It's a useful template for any organization trying to get real value out of AI: what activity metrics miss, how to judge AI's contribution to engineering work, and when it's smarter not to use the tool at all.
What does Amazon's memo say about AI use?
Amazon's memo doesn't argue for avoiding AI. It says the opposite: AI is becoming central to software engineering, but the right practices are still being figured out. The key distinction is between using a capability because it helps and treating adoption itself as the "win."
Amazon said token usage, AI adoption rates, code generation, and code reviews are "deeply flawed" ways to judge teams or organizations if you look at them in isolation. Those metrics can drive up costs and add friction. Amazon didn't say ignore them completely - just that they shouldn't be treated as the definition of success.
The difference between usage and value
A token is a unit of text processed or generated by an AI model. Counting tokens can tell you something about consumption and cost, but it won't tell you whether the output solved a real problem, whether it was correct, or whether the work would have taken longer without the tool.
Adoption tells a similar story. A high percentage of employees using a tool can mean access is widespread - but it doesn't mean people are using it well, that it fits their workflows, or that it's improving outcomes. More generated code can even mean more review, rework, and maintenance if it doesn't match the team's standards.
Amazon's stated preference is to measure long-term outcomes. Marc Brooker, an AWS vice president and distinguished engineer, talked about "feature velocity" - how many launches a team makes and how many successful changes reach production. The framing is straightforward: invest in AI to solve meaningful customer or business problems, not to rack up tokens for their own sake.
What the memo does not claim
Amazon isn't offering a universal measurement system, and it isn't claiming AI has already improved every engineering team's output. The memo also doesn't suggest cost is irrelevant. An Amazon spokesperson said the company hadn't imposed employee token caps. At the same time, Amazon shut down an internal leaderboard after it found employees were tokenmaxxing to climb the rankings.
The useful takeaway is more specific than "measure AI better." Use activity and consumption data as diagnostic signals, not as proof that productivity improved. You still need to decide what outcome you want, what counts as improvement, and what costs or risks you'll accept.
Why is tokenmaxxing a weak productivity strategy?
Tokenmaxxing is what happens when a proxy becomes the target. If an organization rewards high AI usage, people will naturally push usage higher - whether or not the underlying work gets better. That response isn't automatically about bad intent. It's what you'd expect when the measurement system values visible activity more than results you can't measure as easily.
This dynamic shows up well beyond AI. Reward lines of code and you get more code. Reward tickets closed and you get smaller tickets. The metric moves in the "right" direction while the real outcome stays flat - or slips in quality.
It can increase cost without reducing work
AI consumption costs real money and time, even if it shows up in different places: model usage charges, infrastructure spend, and the time developers spend prompting, evaluating, integrating, testing, and fixing output. If someone uses a model for something they could have handled in a few minutes, the tool may add expense without removing meaningful effort.
And the cost calculation isn't only the model bill. A realistic estimate includes the engineer's time across the whole loop: prompts, assessment, integration, testing, and ongoing maintenance. If the AI output takes multiple rounds of correction, the supposed speedup can evaporate. A team that only tracks tokens can miss that, because the token spend is visible while the extra review work is easy to overlook.
It can generate complexity instead of progress
More code doesn't automatically mean more product value. Generated changes still have to fit your existing architecture, handle edge cases, and stay understandable for the people maintaining the system. If an AI-assisted change introduces extra abstractions or dependencies that don't solve a real need, you may get higher short-term output while paying a maintenance bill later.
That pattern matters even more in large organizations. Small "local" choices compound. A bit of unnecessary complexity can be tolerable once. Repeated across services and teams, it makes systems harder to change and incidents harder to diagnose.
It can distort learning and reporting
A leaderboard can turn experimentation into competition. People may pick tasks that are easy to inflate, use AI where it doesn't really fit, and report activity instead of impact. Managers then end up with a misleading picture of where the tool genuinely helps.
That doesn't mean experimentation should stop. Amazon's memo explicitly leaves room for it, including the idea that teams may need a willingness to test - because it's not always obvious upfront which problems justify investment. The difference is between a bounded experiment with a clear question and an open-ended usage push intended to look impressive.
How should teams measure AI's business value?
Start with the work, not the tool. Before you drop an AI assistant into a workflow, define the problem you're trying to improve. "Increase AI adoption" isn't an outcome. "Reduce the time to safely deliver a particular class of change" is closer, because it connects the intervention to something the organization already cares about.
Amazon's reference to feature velocity is helpful because it points toward completed, successful changes - not just generated output. Still, it's not the whole story. A team can ship more changes by lowering quality or splitting work into smaller releases. Whatever you choose as your signal, pair it with quality, reliability, and cost measures that match the product.
Use a four-part measurement model
A practical approach is to compare AI-assisted work against a reasonable baseline across four dimensions:
- Delivery: Did the team complete the work sooner, or deliver more valuable changes in the same time?
- Quality: Did the change pass review and testing, and did it avoid regressions or extra rework?
- Total effort and cost: Include model usage, setup, review, correction, and maintenance - not only the token bill.
- Customer or business outcome: Did the release improve customer experience, reduce operational friction, or address a real business need?
Not every task needs a complicated experiment. For a recurring workflow, set a baseline and compare a sample of similar tasks with and without AI. Keep the comparison honest: account for differences in task complexity, developer experience, and review requirements. If the tasks aren't comparable, treat the result as directional, not definitive.
Make the decision at the right level
A tool can be useful even if it doesn't help every task. AI might help with test scaffolding or exploring an unfamiliar codebase, while providing little benefit for a high-risk change that requires deep domain knowledge. If you only look at a team-wide average, you'll miss those differences.
Track results at the workflow or task level first. Then decide whether the evidence is strong enough to change team practices. Token use can help explain unexpected costs. Adoption can tell you whether people have access - or whether they're trying the tool. Neither of those should replace evidence that the work actually improved.
A simple review after a pilot should answer the basics: What problem did we test? What changed in delivery, quality, and total effort? What did it cost? Would we repeat it, narrow it to certain tasks, or stop? That's how you build a feedback loop rather than running a permanent AI program justified by its own activity.
Key takeaways
- Measure completed outcomes, not raw AI activity.
- Treat tokens and adoption as context, not proof of productivity.
- Include review, rework, and maintenance in the cost of AI-assisted work.
- Keep experiments tied to a specific problem and a decision the results can inform.
When should engineers use AI, and when should they not?
AI is most useful when it reduces meaningful effort without making the output harder to trust or maintain. That can include generating a first draft, summarizing unfamiliar material, or helping explore routine implementation options. The fit depends on the task, the tooling available, and what's at stake if something goes wrong - no single category is always a good use case.
Amazon's memo makes an important point here: if current AI tools don't help in someone's work, that's a finding, not a failure. An engineer who can explain where AI doesn't work is practicing good judgement, not resisting change. This matters because blanket adoption targets can push people to use a tool even when the added review burden outweighs any benefit.
Use AI when the task and risk make sense
Before using an AI tool, ask four questions:
- Is the task well defined? The clearer the expected result, the easier it is to judge whether the output is useful.
- Can a person verify the result? If correctness is hard to check, speed can hide risk.
- What happens if the output is wrong? A reversible draft has a different risk profile than a production change with customer or security implications.
- Does the tool reduce total effort? Count prompting, review, and corrections - not just the first response.
These answers can change across a single project. AI might be helpful for exploring options and drafting tests, while the final design decision and production review remain human responsibilities. The goal isn't to preserve work for people or automate everything you can. It's to put effort where it produces the best result.
Record non-use as operational knowledge
When a team decides not to use AI, record why. Maybe the task is too sensitive, the output can't be verified efficiently, the tool lacks the context it needs, or the workflow is already fast and reliable. That explanation can help future teams avoid repeating the same experiment without learning anything.
Revisit the decision when the task changes or the tools improve. "AI doesn't help here" is useful for the current conditions, not necessarily a permanent rule. The same discipline applies in the other direction: if AI helped once, don't assume it will help every similar task without checking.
Conclusion
Amazon's memo is a useful correction to the idea that AI adoption can be measured by how much AI a company consumes. Token counts and adoption rates can help you understand what's happening, but they don't establish whether the work matters, whether it improved outcomes, or whether the cost was worth it.
For engineering leaders, the practical move is to start by defining the problem, test AI in a bounded workflow, and measure delivery alongside quality, total effort, and business impact. Keep the option to stop or limit use when the evidence doesn't support it. The goal isn't maximum AI usage. It's better work, delivered in a way you can sustain.