No. 003 · Wing of the Global Push

Channel File 291

· hours to fix, days to weeks to clean up

  1. 1Security vendor ships a routine content update to its Windows sensor
  2. 2Update defines more input fields than the sensor code expects
  3. 3Kernel driver reads out of bounds and crashes
  4. 4About 8.5 million Windows machines boot-loop on a blue screen
  5. 5Airlines, hospitals and banks fix machines one by one, by hand ✕

Content updates are code updates. Stage them like code.

Source: CrowdStrike root cause analysis, August 2024

CrowdStrike outage, July 2024. Duration: hours to fix, days to weeks to clean up. 5 hops from the first change to the failure.