Overloaded AWS Transit Gateway takes Slack down on the first workday of 2021
Jan 4, 14:57 UTCJan 4, 17:15 UTC
Duration
2h 18m
Impact
Critical
Root cause
Capacity
Slack, 90 days
7 incidents
Affected
MessagingWeb tierGlobal
Lesson: Managed network components have scaling limits too; pre-warm for known traffic spikes and make sure provisioning survives the incident it is meant to fix.
What happened
One of the AWS Transit Gateways linking Slack VPCs became overloaded as traffic returned after the holidays. Packet loss saturated the web tier, autoscaling tried to add 1,200 servers, and the provisioning service itself overloaded before the web tier recovered.
More from Slack
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 1211:04 UTC | Some customers experiencing blurry image previews and download issues | minor | 8h 56m |
| Sep 1000:34 UTC | Trouble loading Workspace-level Apps & Workflows settings page is returning blank pages | minor | 2h 40m |
| Aug 1321:21 UTC | Trouble Accessing Historical Messages With Custom Data Retention Policies Enabled | minor | 32d 1h |
| Aug 622:22 UTC | Trouble Using Search Bar For Some Admins | minor | 2h 44m |
| Jul 2802:13 UTC | Feature Degradation Affecting the Receipt of Emails in Slack | minor | 1h 30m |
| Jul 2408:22 UTC | Issue sending messages to channels via certain Workflows | minor | 2h 24m |
Also caused by capacity and load
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 2213:19 UTC |  ClickHouse | We are investigating an issue with CH servers in Azure germanywestcentral regionClickHouse | minor | 5h 6m |
| Sep 1607:20 UTC |  GitHub | Degradation with Gemini 3.8 FlashGitHub | major | 10h 28m |
| Sep 1509:47 UTC | GitHub | Disruption with some GitHub servicesGitHub | minor | 1h 30m |
| Sep 422:02 UTC | GitHub | Degradation in repos contents APIGitHub | minor | 21m |
| Sep 221:44 UTC |  Expo | Elevated Linux worker queue timesExpo | minor | 21m |
| Sep 115:00 UTC | GitHub | Delays in commit processingGitHub | minor | 1h 1m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.