DynamoDB DNS failure takes down US-EAST-1
Oct 20, 06:48 UTCOct 20, 21:20 UTC
Duration
14h 32m
Impact
Critical
Root cause
DNS
AWS, 90 days
12 incidents
Affected
DynamoDBEC2Network Load BalancerLambdaus-east-1
Lesson: Automation that manages critical DNS needs its own guard against writing an empty record, and dependent services need to recover from a stale state on their own.
What happened
A latent race condition in the DynamoDB DNS management system left an empty DNS record for the regional endpoint dynamodb.us-east-1.amazonaws.com, and the automation failed to repair it. EC2 launches and Network Load Balancer health checks failed in turn, giving three distinct periods of customer impact.
More from AWS
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2123:26 UTC | Increased Error Rates | minor | 58m |
| Sep 321:49 UTC | Increased API Error Rates | minor | 2h 18m |
| Aug 2102:02 UTC | Increased Error Rates | minor | 38m |
| Aug 1915:15 UTC | Increased Error Rates | minor | 3h 32m |
| Aug 1503:42 UTC | Increased Packet loss | minor | 3d |
| Jul 3117:33 UTC | Elevated Packet Loss | minor | 1h 21m |
Also caused by dns
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Jul 2720:34 UTC |  Supabase | Delays in creation of DNS recordsSupabase | minor | 2h 2m |
| Jun 2301:05 UTC |  Snowflake | INC20000046Snowflake | major | 1h 30m |
| Jun 410:12 UTC |  DigitalOcean | DNS API ServiceDigitalOcean | minor | 1h 35m |
| May 2223:27 UTC |  Upstash | New Database Creation Failing Due to Upstream Provider IssueUpstash | none | 32m |
| May 1413:57 UTC | DigitalOcean | DNS service, Certificates and Managed MongoDBDigitalOcean | minor | 6h 49m |
| May 715:17 UTC |  MongoDB | AWS Cluster DNS Changes Degraded PerformanceMongoDB | major | 6h 58m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.