Connectivity Issues in us-east-1
Aug 18, 12:37 UTCAug 18, 12:37 UTC
Duration
0m
Impact
Minor
Root cause
Config change
Upstash, 90 days
5 incidents
Affected
N. Virginia, USA (us-east-1)Redis Global - N. Virginia, USA (us-east-1)
Final update
Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial phase of a gradual deployment. Upon detection, the deployment was immediately rolled back, restoring normal service. We are reviewing our deployment and configuration validation processes to prevent similar issues in the future.
Timeline
- Postmortem · Aug 18, 13:30 UTC
Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial phase of a gradual deployment. Upon detection, the deployment was immediately rolled back, restoring normal service. We are reviewing our deployment and configuration validation processes to prevent similar issues in the future.
- Resolved · Aug 18, 12:37 UTC
Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial phase of a gradual deployment. Upon detection, the deployment was immediately rolled back, restoring normal service. We are reviewing our deployment and configuration validation processes to prevent similar issues in the future.
More from Upstash
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 913:00 UTC | Intermittent DNS Resolution Errors for Upstash Vector in US East (us-east-1) | minor | 0m |
| Sep 308:29 UTC | Upstash Console login issue | none | 0m |
| Aug 2822:56 UTC | Message Persistence Issue , QStash us-east-1 | major | 0m |
| Jul 2208:50 UTC | Fly.io infrastructure disruption affecting some Upstash Redis databases on Fly.io DFW Region | none | 1h 15m |
| Jul 1608:35 UTC | QStash us-east-1 - URL Publish Errors | none | 0m |
| Jun 2515:23 UTC | Upstash Redis Partial Service Disruption | major | 1h 12m |
Also caused by configuration change
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 2218:20 UTC |  Twilio | Phone Number APIs and Console Were Returning Incorrect 404 ResponsesTwilio | none | 0m |
| Sep 1622:30 UTC |  Cloudflare | Hyperdrive Elevated Origin Connection Failure RatesCloudflare | none | 0m |
| Sep 1211:04 UTC |  Slack | Some customers experiencing blurry image previews and download issuesSlack | minor | 8h 56m |
| Sep 1107:18 UTC |  Snowflake | INC20000213Snowflake | critical | 6h 1m |
| Sep 407:34 UTC | Snowflake | INC20000199Snowflake | critical | 2h 36m |
| Sep 109:58 UTC | Snowflake | INC20000190Snowflake | major | 3h 52m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.