Mistyped command removes S3 index servers in US-EAST-1
Feb 28, 17:37 UTCFeb 28, 21:54 UTC
Duration
4h 17m
Impact
Critical
Root cause
Operator
AWS, 90 days
12 incidents
Affected
S3Service Health Dashboardus-east-1
Lesson: Tools should refuse to remove capacity below a safe minimum, and the status page cannot depend on the system it reports on.
What happened
An S3 team member ran an established playbook command with one input entered incorrectly, removing far more servers than intended, including those behind the index and placement subsystems. Both needed full restarts; the status dashboard itself depended on S3.
More from AWS
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2123:26 UTC | Increased Error Rates | minor | 58m |
| Sep 321:49 UTC | Increased API Error Rates | minor | 2h 18m |
| Aug 2102:02 UTC | Increased Error Rates | minor | 38m |
| Aug 1915:15 UTC | Increased Error Rates | minor | 3h 32m |
| Aug 1503:42 UTC | Increased Packet loss | minor | 3d |
| Jul 3117:33 UTC | Elevated Packet Loss | minor | 1h 21m |
Also caused by operator action
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Feb 608:14 UTC |  Cloudflare | R2 object storage disabled during a phishing report remediationCloudflare | major | 1h 22m |
| Apr 507:38 UTC |  Atlassian | Maintenance script deletes 883 customer sitesAtlassian | critical | 12d 16h |
| Jan 3123:00 UTC |  GitLab | Primary database data accidentally deleted, 18-hour restoreGitLab | critical | 19h |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.