No. 006 · The Enter Key Gallery
The Typo That Took Down S3
· about 4 hours
Impact 4.8/10
- 1Engineer debugging a slow billing system runs an approved playbook command
- 2One input is mistyped; far more servers are removed than intended
- 3Two S3 subsystems in us-east-1 need a full restart
- 4They haven't been fully restarted in years; it takes hours
- 5A large part of the web goes down, including AWS's own status dashboard ✕
Tools should refuse to remove too much capacity too fast, however it's typed.
Source: AWS service disruption summary, March 2017
Impact
Five dimensions, 0 to 10 each. The overall 4.8/10 is their average. How impact is scored
| Dimension | Score | Basis |
|---|
| Reach | 4 | S3 in one region, us-east-1 |
|---|
| Duration | 6 | About 4 hours |
|---|
| Loss | 0 | None stated in the source |
|---|
| Recovery | 6 | Two subsystems needed a full restart, the first in years |
|---|
| Cascade | 8 | A large part of the web, AWS's own status dashboard included |
|---|
AWS S3 outage, February 2017. Duration: about 4 hours. 5 hops from the first change to the failure.
See this exhibit in the 3D hall