No. 005 · Wing of the Global Push
One Customer, One Setting
· under an hour
Impact 4.0/10
- 1A software change deployed weeks earlier carries a latent bug
- 2One customer pushes a valid configuration change
- 3The change triggers the bug across the edge network
- 4Most of the network starts returning errors; major news sites go blank
- 5Engineers find the config, disable it, and traffic recovers ✕
A bug can wait patiently for the one input that wakes it.
Source: Fastly blog, June 2021
Impact
Five dimensions, 0 to 10 each. The overall 4.0/10 is their average. How impact is scored
| Dimension | Score | Basis |
|---|
| Reach | 8 | Most of the edge network, worldwide |
|---|
| Duration | 4 | 95% of the network back within 49 minutes |
|---|
| Loss | 0 | None stated in the source |
|---|
| Recovery | 2 | The customer configuration was disabled |
|---|
| Cascade | 6 | Customer sites went blank, major news sites among them |
|---|
Fastly outage, June 2021. Duration: under an hour. 5 hops from the first change to the failure.
See this exhibit in the 3D hall