No. 001 · Hall of Lost Packets

The Day Facebook Left the Internet

· about 6 hours

Impact 4.8/10

  1. 1Routine maintenance command meant to assess backbone capacity
  2. 2Command takes down every backbone connection between data centers
  3. 3DNS servers see the backbone is gone and withdraw their BGP routes
  4. 4facebook.com, Instagram and WhatsApp stop resolving worldwide
  5. 5Engineers need physical access to reset routers; internal tools are down too ✕

If your recovery tools live behind the thing that broke, you don't have recovery tools.

Source: Facebook Engineering blog, October 2021

Impact

Five dimensions, 0 to 10 each. The overall 4.8/10 is their average. How impact is scored

Reach8Duration6Loss0Recovery8Cascade2
DimensionScoreBasis
Reach8Facebook, Instagram and WhatsApp, worldwide
Duration6About 6 hours
Loss0None stated in the source
Recovery8Engineers had to reach the data centers in person
Cascade2Internal tools went down with it

Facebook outage, October 2021. Duration: about 6 hours. 5 hops from the first change to the failure.

See this exhibit in the 3D hall