Customer configuration triggers latent bug, 85% of network errors
Jun 8, 09:47 UTCJun 8, 12:35 UTC
Duration
2h 48m
Impact
Critical
Root cause
Bug
Fastly, 90 days
0 incidents
Affected
CDNGlobal
Lesson: Customer configuration is untrusted input to a shared fleet; isolate its blast radius and test the bug classes it can reach.
What happened
A software deployment on May 12 introduced a bug that a specific customer configuration could trigger. On June 8 a customer pushed a valid configuration change that did, and 85% of the network returned errors. Most services recovered by 11:00 UTC; the incident was mitigated at 12:35 UTC.
Also caused by software bug
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1518:57 UTC |  ClickHouse | ClickPipes failing on Kinesis in AWS us-east-1ClickHouse | critical | 29h 46m |
| Sep 318:20 UTC |  Twilio | Retroactive Incident: Twilio Personalized Support Phone Line AffectedTwilio | none | 0m |
| Aug 819:48 UTC |  Grafana Labs | Alerting expressions pipeline failing when recovery settingsGrafana Labs | minor | 0m |
| Aug 615:22 UTC |  GitHub | Incident with ActionsGitHub | critical | 10h 42m |
| Jul 2314:14 UTC |  Box | [Medium] Issues with Box HubsBox | major | 16m |
| Jun 1719:00 UTC | GitHub | Incident With WebhooksGitHub | none | 0m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.