No. 005 · Wing of the Global Push

One Customer, One Setting

· under an hour

Impact 4.0/10

  1. 1A software change deployed weeks earlier carries a latent bug
  2. 2One customer pushes a valid configuration change
  3. 3The change triggers the bug across the edge network
  4. 4Most of the network starts returning errors; major news sites go blank
  5. 5Engineers find the config, disable it, and traffic recovers ✕

A bug can wait patiently for the one input that wakes it.

Source: Fastly blog, June 2021

Impact

Five dimensions, 0 to 10 each. The overall 4.0/10 is their average. How impact is scored

Reach8Duration4Loss0Recovery2Cascade6
DimensionScoreBasis
Reach8Most of the edge network, worldwide
Duration495% of the network back within 49 minutes
Loss0None stated in the source
Recovery2The customer configuration was disabled
Cascade6Customer sites went blank, major news sites among them

Fastly outage, June 2021. Duration: under an hour. 5 hops from the first change to the failure.

See this exhibit in the 3D hall