Amazon Memo Calls Token Usage a 'Deeply Flawed' Metric, Warns Against 'Tokenmaxxing'

2.7 · 4d ago · US · AI

The Facts

  • Amazon posted a memo on its website on Tuesday laying out AI usage guidelines for software engineers; it was circulated internally in September. CNBC only
  • The memo says metrics such as token usage, AI tool adoption rate, and code changes or reviews are 'deeply flawed' ways to measure teams' or organizations' AI usage, and can lead to higher costs and too much complexity. CNBC only
  • The memo does not tell staff to ignore the metrics entirely; token usage and code generation should be considered 'with care and judgment' alongside other data. CNBC only
  • The memo states that work not benefiting from current AI tooling 'is a finding, not a failing.' CNBC only
  • An Amazon spokesperson said the company has not instituted caps on employees' token usage. CNBC only
Dive deeper · 3 more factsShow less
  • Amazon shut down an internal leaderboard in May after finding employees were tokenmaxxing to climb the ranks. CNBC only
  • AWS vice president and distinguished engineer Marc Brooker said Amazon uses metrics such as 'feature velocity' (launches and successful production changes) to gauge AI's effect on engineer productivity. CNBC only
  • Other companies have set limits on employee token usage or switched to cheaper open-source models for some tasks to curb rising AI costs. CNBC only

Context

Amazon published guidelines on how software engineers and managers should use AI, discouraging 'tokenmaxxing', the practice of using AI as much as possible with little regard for output. The memo favors measuring long-term outcomes over usage statistics.

CEO Andy Jassy has said AI will change the scope and speed of work and could shrink the corporate workforce in coming years, while also creating other jobs. Companies broadly are grappling with rising AI costs and how to measure returns.

How the Agent Panel Scored This

Independent AI personas each scored importance 1–10; the story's score is a trimmed mean (highest and lowest votes dropped). Vote spread: 1.

3 Business Analyst This is an internal AI usage guideline memo from Amazon that discourages certain metrics, with no new binding policy, caps, or material financial action beyond a spokesperson's statement that no caps exist, so its economic impact is limited to a minor corporate-policy signal.
3 News Editor This story reports an internal-policy memo on AI usage metrics at one company, a concrete but narrow corporate guideline with limited direct impact on most people beyond Amazon engineers.
2 Informed Citizen This is an internal corporate guidelines memo about how engineers should measure AI usage, with no broad impact on ordinary people and limited relevance beyond the tech industry and its staff.

Source

← All stories