Oversized Bot Management feature file breaks the core proxy
Nov 18, 11:20 UTCNov 18, 17:06 UTC
Duration
5h 46m
Impact
Critical
Root cause
Config change
Cloudflare, 90 days
50 incidents
Affected
CDNBot ManagementWorkers KVAccessTurnstileDashboardGlobal
Lesson: Treat internally generated config files like user input: validate size and shape before they propagate, and keep global kill switches for each feature.
What happened
A change to database permissions made a ClickHouse query return duplicate rows, doubling the Bot Management feature file. The larger file exceeded a limit in the core proxy, which returned HTTP 5xx errors across the network. Core traffic was largely normal by 14:30 UTC and all systems by 17:06 UTC.
More from Cloudflare
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:42 UTC | Network Performance Degradation , Asia-Pacific | minor | Ongoing |
| Sep 2315:54 UTC | Elevated Errors with any / all in http_response_cache_settings | minor | 2h 14m |
| Sep 2309:08 UTC | Intermittent authentication errors for API and R2 | minor | 11h 51m |
| Sep 2308:09 UTC | Increased Errors for Durable Objects | minor | 1h 51m |
| Sep 2304:08 UTC | Elevated number of R2 503 errors in Australian Eastern Coast region | minor | 17m |
| Sep 2300:42 UTC | Issues with 1.1.1.1 for Families | minor | 44m |
Also caused by configuration change
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 2218:20 UTC |  Twilio | Phone Number APIs and Console Were Returning Incorrect 404 ResponsesTwilio | none | 0m |
| Sep 1211:04 UTC |  Slack | Some customers experiencing blurry image previews and download issuesSlack | minor | 8h 56m |
| Sep 1107:18 UTC |  Snowflake | INC20000213Snowflake | critical | 6h 1m |
| Sep 407:34 UTC | Snowflake | INC20000199Snowflake | critical | 2h 36m |
| Sep 109:58 UTC | Snowflake | INC20000190Snowflake | major | 3h 52m |
| Sep 108:01 UTC |  Sentry | Sentry requests timing out / 504'sSentry | major | 3h 20m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.