No. 004 · Wing of the Global Push

One Regular Expression

· 27 minutes

Impact 3.6/10

  1. 1New managed WAF rule is deployed globally at once
  2. 2Its regular expression backtracks catastrophically on ordinary traffic
  3. 3CPU on every edge machine hits 100%
  4. 4Sites behind the network return 502s worldwide
  5. 5Global kill switch for the WAF rules restores traffic ✕

The fastest deploy pipeline is also the fastest outage pipeline.

Source: Cloudflare blog, July 2019

Impact

Five dimensions, 0 to 10 each. The overall 3.6/10 is their average. How impact is scored

Reach8Duration2Loss0Recovery2Cascade6
DimensionScoreBasis
Reach8Sites behind Cloudflare, worldwide
Duration227 minutes
Loss0None stated in the source
Recovery2A global kill switch for the WAF rules
Cascade6Customer sites returned 502 errors

Cloudflare regex outage, July 2019. Duration: 27 minutes. 5 hops from the first change to the failure.

See this exhibit in the 3D hall