Upstash · Data and observabilityJun 11, 2025, 06:51 UTC

Degraded Performance

CriticalNot disclosedUpdated 20h ago

Jun 11, 06:51 UTCJun 11, 10:02 UTC

Duration

3h 11m

Impact

Critical

Root cause

Not disclosed

Upstash, 90 days

5 incidents

Affected

N. Virginia, USA (us-east-1)N. California, USA (us-west-1)Oregon, USA (us-west-2)Frankfurt, Germany (eu-central-1)Ireland (eu-west-1)Singapore (ap-southeast-1)Sydney, Australia (ap-southeast-2)Mumbai, India (ap-south-1)Tokyo, Japan (ap-northeast-1)São Paulo, Brazil (sa-east-1)Ohio, USA (us-east-2)London, UK (eu-west-2)Redis Global - N. Virginia, USA (us-east-1)Redis Global - N. California, USA (us-west-1)

Status page

Final update

A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts. As a result, several EC2 nodes failed health checks and temporarily dropped out of the cluster, disrupting high availability and causing partial connectivity issues for some clients and operations. We have since reproduced the issue in a controlled environment and verified the root cause. To prevent a recurrence, we are updating our node maintenance strategy to ensure greater control over the timing and impact of system-level changes and excluding networking components from automated upgrades.

Timeline

Postmortem · Jun 11, 11:49 UTC
A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts. As a result, several EC2 nodes failed health checks and temporarily dropped out of the cluster, disrupting high availability and causing partial connectivity issues for some clients and operations. We have since reproduced the issue in a controlled environment and verified the root cause. To prevent a recurrence, we are updating our node maintenance strategy to ensure greater control over the timing and impact of system-level changes and excluding networking components from automated upgrades.
Resolved · Jun 11, 10:02 UTC
This incident has been resolved.
Monitoring · Jun 11, 09:20 UTC
A fix has been implemented and we are monitoring the results.
Investigating · Jun 11, 08:24 UTC
We are continuing to investigate this issue.
Investigating · Jun 11, 06:51 UTC
We are currently investigating this issue.

More from Upstash

Full history

Started	Incident	Impact	Duration
Sep 913:00 UTC	Intermittent DNS Resolution Errors for Upstash Vector in US East (us-east-1)	minor	0m
Sep 308:29 UTC	Upstash Console login issue	none	0m
Aug 2822:56 UTC	Message Persistence Issue , QStash us-east-1	major	0m
Jul 2208:50 UTC	Fly.io infrastructure disruption affecting some Upstash Redis databases on Fly.io DFW Region	none	1h 15m
Jul 1608:35 UTC	QStash us-east-1 - URL Publish Errors	none	0m
Jun 2515:23 UTC	Upstash Redis Partial Service Disruption	major	1h 12m

From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.