No. 007 · The Enter Key Gallery
rm -rf, Wrong Terminal
· about 18 hours; roughly 6 hours of data lost
Impact 5.2/10
- 1Late-night fight with database replication lag
- 2Engineer deletes a data directory to reseed a replica
- 3The shell was on the primary, not the replica
- 4The backup methods turn out to be broken or empty
- 5Restore from a staging snapshot, livestreamed to the public ✕
A backup you have never restored is a hope, not a backup.
Source: GitLab postmortem, February 2017
Impact
Five dimensions, 0 to 10 each. The overall 5.2/10 is their average. How impact is scored
| Dimension | Score | Basis |
|---|
| Reach | 6 | GitLab.com users worldwide |
|---|
| Duration | 8 | About 18 hours |
|---|
| Loss | 6 | About 6 hours of database data lost |
|---|
| Recovery | 6 | Restored from a staging snapshot |
|---|
| Cascade | 0 | Contained to GitLab.com |
|---|
GitLab database incident, January 2017. Duration: about 18 hours; roughly 6 hours of data lost. 5 hops from the first change to the failure.
See this exhibit in the 3D hall