No. 007 · The Enter Key Gallery

rm -rf, Wrong Terminal

· about 18 hours; roughly 6 hours of data lost

Impact 5.2/10

  1. 1Late-night fight with database replication lag
  2. 2Engineer deletes a data directory to reseed a replica
  3. 3The shell was on the primary, not the replica
  4. 4The backup methods turn out to be broken or empty
  5. 5Restore from a staging snapshot, livestreamed to the public ✕

A backup you have never restored is a hope, not a backup.

Source: GitLab postmortem, February 2017

Impact

Five dimensions, 0 to 10 each. The overall 5.2/10 is their average. How impact is scored

Reach6Duration8Loss6Recovery6Cascade0
DimensionScoreBasis
Reach6GitLab.com users worldwide
Duration8About 18 hours
Loss6About 6 hours of database data lost
Recovery6Restored from a staging snapshot
Cascade0Contained to GitLab.com

GitLab database incident, January 2017. Duration: about 18 hours; roughly 6 hours of data lost. 5 hops from the first change to the failure.

See this exhibit in the 3D hall