No. 001 · Hall of Lost Packets
The Day Facebook Left the Internet
· about 6 hours
Impact 4.8/10
- 1Routine maintenance command meant to assess backbone capacity
- 2Command takes down every backbone connection between data centers
- 3DNS servers see the backbone is gone and withdraw their BGP routes
- 4facebook.com, Instagram and WhatsApp stop resolving worldwide
- 5Engineers need physical access to reset routers; internal tools are down too ✕
If your recovery tools live behind the thing that broke, you don't have recovery tools.
Source: Facebook Engineering blog, October 2021
Impact
Five dimensions, 0 to 10 each. The overall 4.8/10 is their average. How impact is scored
| Dimension | Score | Basis |
|---|
| Reach | 8 | Facebook, Instagram and WhatsApp, worldwide |
|---|
| Duration | 6 | About 6 hours |
|---|
| Loss | 0 | None stated in the source |
|---|
| Recovery | 8 | Engineers had to reach the data centers in person |
|---|
| Cascade | 2 | Internal tools went down with it |
|---|
Facebook outage, October 2021. Duration: about 6 hours. 5 hops from the first change to the failure.
See this exhibit in the 3D hall