Maintenance script deletes 883 customer sites
Apr 5, 07:38 UTCApr 18, 00:00 UTC
Duration
12d 16h
Impact
Critical
Root cause
Operator
Atlassian, 90 days
0 incidents
Affected
JiraConfluenceOpsgenieGlobal
Lesson: Deletion should be soft by default, and bulk restores need to be rehearsed at the scale of your largest possible mistake.
What happened
A script meant to deactivate a legacy app was given the IDs of entire cloud sites and run in permanent delete mode, deleting 883 sites belonging to 775 customers between 07:38 and 08:01 UTC. Some customers were down for up to 14 days while sites were restored one by one.
More from Atlassian
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Mar 407:31 UTC | Issues with 403 user authentication errors across Atlassian products | none | 0m |
| Sep 1010:26 UTC | Slow loading times and extended response times for Cloud products in the EU-West region. | none | 2h 18m |
| Sep 908:56 UTC | High RDS CPU on multiple environments | none | 9h 16m |
| Sep 209:23 UTC | Slowness in Jira. | none | 8d 3h |
| Aug 2011:46 UTC | Data residency migrations halted. | minor | 2d 23h |
| Jul 320:51 UTC | Some products are hard down | none | 7h 3m |
Also caused by operator action
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Feb 608:14 UTC |  Cloudflare | R2 object storage disabled during a phishing report remediationCloudflare | major | 1h 22m |
| Feb 2817:37 UTC |  AWS | Mistyped command removes S3 index servers in US-EAST-1AWS | critical | 4h 17m |
| Jan 3123:00 UTC |  GitLab | Primary database data accidentally deleted, 18-hour restoreGitLab | critical | 19h |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.