WAF regular expression exhausts CPU worldwide for 27 minutes
Jul 2, 13:42 UTCJul 2, 14:09 UTC
Duration
27m
Impact
Critical
Root cause
Deployment
Cloudflare, 90 days
50 incidents
Affected
CDNWAFGlobal
Lesson: Rules and regexes are code; stage them, and use an engine with guaranteed linear time for untrusted input.
What happened
A new WAF rule for XSS detection contained a regular expression with catastrophic backtracking. Deployed globally through an automatic process, it pushed CPU to 100% on every HTTP server and the network dropped traffic for 27 minutes.
More from Cloudflare
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:42 UTC | Network Performance Degradation , Asia-Pacific | minor | Ongoing |
| Sep 2315:54 UTC | Elevated Errors with any / all in http_response_cache_settings | minor | 2h 14m |
| Sep 2309:08 UTC | Intermittent authentication errors for API and R2 | minor | 11h 51m |
| Sep 2308:09 UTC | Increased Errors for Durable Objects | minor | 1h 51m |
| Sep 2304:08 UTC | Elevated number of R2 503 errors in Australian Eastern Coast region | minor | 17m |
| Sep 2300:42 UTC | Issues with 1.1.1.1 for Families | minor | 44m |
Also caused by deployment or rollout
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Aug 2014:43 UTC |  GitHub | Intermittent failures creating agent tasksGitHub | critical | 9h 54m |
| Aug 2000:31 UTC |  Box | [Medium] Issue with Microsoft Office Integration and Box EditBox | major | 1h 13m |
| Aug 622:22 UTC |  Slack | Trouble Using Search Bar For Some AdminsSlack | minor | 2h 44m |
| Jul 2110:23 UTC |  dbt Labs | Job runs failing at the git clone stepdbt Labs | critical | 2h 29m |
| Jul 1907:33 UTC | dbt Labs | Some EMEA customers experiencing Git clone failures (403 errors) on thier accountsdbt Labs | major | 6h 22m |
| Jul 200:17 UTC |  Sentry | Traces, spans, logs inaccesible for query or ingestion in de and usSentry | none | 3h 39m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.