No. 006 · The Enter Key Gallery

The Typo That Took Down S3

· about 4 hours

Impact 4.8/10

  1. 1Engineer debugging a slow billing system runs an approved playbook command
  2. 2One input is mistyped; far more servers are removed than intended
  3. 3Two S3 subsystems in us-east-1 need a full restart
  4. 4They haven't been fully restarted in years; it takes hours
  5. 5A large part of the web goes down, including AWS's own status dashboard ✕

Tools should refuse to remove too much capacity too fast, however it's typed.

Source: AWS service disruption summary, March 2017

Impact

Five dimensions, 0 to 10 each. The overall 4.8/10 is their average. How impact is scored

Reach4Duration6Loss0Recovery6Cascade8
DimensionScoreBasis
Reach4S3 in one region, us-east-1
Duration6About 4 hours
Loss0None stated in the source
Recovery6Two subsystems needed a full restart, the first in years
Cascade8A large part of the web, AWS's own status dashboard included

AWS S3 outage, February 2017. Duration: about 4 hours. 5 hops from the first change to the failure.

See this exhibit in the 3D hall