New telemetry service overwhelms Kubernetes control planes
Dec 11, 23:16 UTCDec 12, 03:38 UTC
Duration
4h 22m
Impact
Critical
Root cause
Deployment
OpenAI, 90 days
26 incidents
Affected
ChatGPTAPISoraGlobal
Lesson: Test changes at production cluster size, and keep break-glass access to the control plane that does not depend on the thing that is failing.
What happened
A new telemetry service made every node run expensive Kubernetes API operations, overwhelming the API servers in the largest clusters. DNS-based service discovery depends on the control plane, so services failed as DNS caches expired, and engineers were locked out of the control plane needed to roll back.
More from OpenAI
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2323:25 UTC | Mobile users unable to see Work Mode and the Model Picker | minor | 24m |
| Sep 2309:13 UTC | Elevated Error Rates for ChatGPT across Plus and Pro plans. | minor | 41m |
| Sep 2215:04 UTC | Elevated Error Rates for ChatGPT Work, across Plus, Pro, Business, Enterprise and Education plans. | minor | 54m |
| Sep 2209:58 UTC | Increased error rate for Plus and Pro users. | minor | 39m |
| Sep 1823:00 UTC | SSO sign-in and SCIM provisioning issues | none | 19m |
| Sep 1822:29 UTC | Overbilling for OpenAI-hosted containers in the Agent API | none | 9h 23m |
Also caused by deployment or rollout
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1106:22 UTC |  Cloudflare | Issues with Workers VPC hostname route resolution on 2026-09-11Cloudflare | none | 0m |
| Aug 2014:43 UTC |  GitHub | Intermittent failures creating agent tasksGitHub | critical | 9h 54m |
| Aug 2000:31 UTC |  Box | [Medium] Issue with Microsoft Office Integration and Box EditBox | major | 1h 13m |
| Aug 622:22 UTC |  Slack | Trouble Using Search Bar For Some AdminsSlack | minor | 2h 44m |
| Jul 2110:23 UTC |  dbt Labs | Job runs failing at the git clone stepdbt Labs | critical | 2h 29m |
| Jul 1907:33 UTC | dbt Labs | Some EMEA customers experiencing Git clone failures (403 errors) on thier accountsdbt Labs | major | 6h 22m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.