Performance degradation on QStash
Mar 12, 09:43 UTCMar 12, 10:16 UTC
Duration
34m
Impact
Minor
Root cause
Not disclosed
Upstash, 90 days
5 incidents
Affected
EU-CENTRAL-1QStash
Final update
On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of network resources and negatively impacted all users. We have added more resources to QStash as a quick remediation and as for the resolution, we have improved the timeout mechanism to detect and fail-faster in such cases.
Timeline
- Postmortem · Mar 12, 12:26 UTC
On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of network resources and negatively impacted all users. We have added more resources to QStash as a quick remediation and as for the resolution, we have improved the timeout mechanism to detect and fail-faster in such cases.
- Resolved · Mar 12, 10:16 UTC
This incident has been resolved.
- Monitoring · Mar 12, 10:08 UTC
A fix has been implemented and we are monitoring the results.
- Investigating · Mar 12, 09:43 UTC
We are currently investigating this issue.
More from Upstash
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 913:00 UTC | Intermittent DNS Resolution Errors for Upstash Vector in US East (us-east-1) | minor | 0m |
| Sep 308:29 UTC | Upstash Console login issue | none | 0m |
| Aug 2822:56 UTC | Message Persistence Issue , QStash us-east-1 | major | 0m |
| Jul 2208:50 UTC | Fly.io infrastructure disruption affecting some Upstash Redis databases on Fly.io DFW Region | none | 1h 15m |
| Jul 1608:35 UTC | QStash us-east-1 - URL Publish Errors | none | 0m |
| Jun 2515:23 UTC | Upstash Redis Partial Service Disruption | major | 1h 12m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.