OS update breaks networking across regions
Mar 8, 06:03 UTCMar 9, 08:58 UTC
Duration
26h 55m
Impact
Critical
Root cause
Deployment
Datadog, 90 days
14 incidents
Affected
Web appIngestionMonitorsAPMUS1US3US5EU1
Lesson: Automatic updates are deployments: stagger them, and never let the same change land on every region in the same hour.
What happened
An automatic operating system security update restarted a networking component on many hosts at once, removing network routes and disconnecting nodes across several regions and cloud providers. All services were operational on March 9 at 08:58 UTC, with historical data backfilled by March 10.
More from Datadog
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2109:44 UTC | Delayed Monitors Notifications | minor | 46m |
| Sep 2005:17 UTC | Delayed events triggered by emails | minor | 1h 36m |
| Sep 317:20 UTC | Delayed CICD Optimization, Code Coverage, Code Security, and DORA data | minor | 5h 41m |
| Sep 201:07 UTC | Delayed DBM Monitor Evaluation | minor | 17m |
| Aug 618:51 UTC | Delayed Processes data | minor | 46m |
| Jul 3007:44 UTC | Delayed Evaluation of Service Check monitors | minor | 43m |
Also caused by deployment or rollout
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1106:22 UTC |  Cloudflare | Issues with Workers VPC hostname route resolution on 2026-09-11Cloudflare | none | 0m |
| Aug 2014:43 UTC |  GitHub | Intermittent failures creating agent tasksGitHub | critical | 9h 54m |
| Aug 2000:31 UTC |  Box | [Medium] Issue with Microsoft Office Integration and Box EditBox | major | 1h 13m |
| Aug 622:22 UTC |  Slack | Trouble Using Search Bar For Some AdminsSlack | minor | 2h 44m |
| Jul 2110:23 UTC |  dbt Labs | Job runs failing at the git clone stepdbt Labs | critical | 2h 29m |
| Jul 1907:33 UTC | dbt Labs | Some EMEA customers experiencing Git clone failures (403 errors) on thier accountsdbt Labs | major | 6h 22m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.