Degraded performance when accessing 1Password

Final update

# Incident Postmortem - Degraded performance when accessing 1Password **Date of Incident:** 2025-09-26 **Time of Incident:** 4:20pm UTC - 5:39pm UTC **Service\(s\) Affected:** SSO, Web Sign In, Sign Up, Web Interface, CLI **Impact Duration:** ~60 minutes ## Summary On September 26, 2025 at 4:20 UTC 1Password’s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected. ## Impact on Customers During the duration of the incident: * **Web interface, Administration:** Customers experienced delays when accessing the 1Password web interface. * **Single Sign-on \(SSO\), Multi-factor Authentication \(MFA\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to login. * **Command Line Interface \(CLI\):** CLI users faced increased latency and timeouts when attempting to access our web APIs. * **Browser Extension:** Users requiring web interface authentication experienced delays or failures. * **Number of Affected Customers \(approximate\):** ~30% * **Geographic Regions Affected:** [1password.com](http://1password.com) \(US/Global\) ## What Happened?

1. Postmortem · Oct 3, 19:24 UTC
   # Incident Postmortem - Degraded performance when accessing 1Password **Date of Incident:** 2025-09-26 **Time of Incident:** 4:20pm UTC - 5:39pm UTC **Service\(s\) Affected:** SSO, Web Sign In, Sign Up, Web Interface, CLI **Impact Duration:** ~60 minutes ## Summary On September 26, 2025 at 4:20 UTC 1Password’s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected. ## Impact on Customers During the duration of the incident: * **Web interface, Administration:** Customers experienced delays when accessing the 1Password web interface. * **Single Sign-on \(SSO\), Multi-factor Authentication \(MFA\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to login. * **Command Line Interface \(CLI\):** CLI users faced increased latency and timeouts when attempting to access our web APIs. * **Browser Extension:** Users requiring web interface authentication experienced delays or failures. * **Number of Affected Customers \(approximate\):** ~30% * **Geographic Regions Affected:** [1password.com](http://1password.com) \(US/Global\) ## What Happened? At 4:20PM UTC and 5 PM UTC There were traffic bursts which caused extra load on one of our caches. This cache was under-provisioned to handle that spike of activity, which resulted in it exhausting available CPU. This caused cascading errors/latency which manifested in slow and failed requests. * *
2. Resolved · Sep 26, 20:24 UTC
   This incident has been resolved. We will publish a post-mortem as soon as we complete it.
3. Monitoring · Sep 26, 19:53 UTC
   The engineering team deployed a mitigation, which has addressed the issue. We will continue to monitor performance.
4. Identified · Sep 26, 19:28 UTC
   The engineering team is rolling out a mitigation in our production environment.
5. Identified · Sep 26, 19:13 UTC
   The engineering team is continuing to test a mitigation in our test environment before deploying it to production. Customer impact is being actively managed and instances of slowdowns should continue to decrease.
6. Identified · Sep 26, 18:57 UTC
   The engineering team has identified the issue and we are continuing to test a mitigation in our test environment before rolling it out to production. Customer impact is being actively managed and slowdowns should be increasingly rare.
7. Investigating · Sep 26, 18:44 UTC
   Engineering teams are testing a mitigation in our non-production environments to verify before rolling it out to our production environments. Customer impact is being actively managed and slowdowns should be rare if they occur at all.
8. Investigating · Sep 26, 18:32 UTC
   Engineering teams have identified a potential cause and are actively managing customer impact. Customers may see some slowdowns, but due to the active management those slowdowns are unlikely.
9. Investigating · Sep 26, 18:11 UTC
   The engineering team is actively managing degraded performance and continuing attempts to identify root cause. Customers may still experience some slowdowns when accessing 1Password online.
10. Investigating · Sep 26, 17:57 UTC
    The engineering team is actively managing degraded performance and simultaneously attempting to identify root cause. Customers may still experience some slowdowns when accessing 1Password online.
11. Investigating · Sep 26, 17:41 UTC
    The engineering team is continuing to investigate and actively managing service degradation. Customers may still experience periods of slowdowns when accessing the 1Password service.
12. Investigating · Sep 26, 17:25 UTC
    Overall service is still degraded, but experiencing periods of improvement.
13. Investigating · Sep 26, 17:06 UTC
    The engineering team is restarting key systems in an attempt to alleviate slowdowns.
14. Investigating · Sep 26, 16:51 UTC
    We are actively investigating an issue where customers may be experiencing degraded performance and slowdowns when accessing 1Password.