Elevated API Errors

Final update

# Performance Degradation **Date of Incident:** 2025-09-03 **Time of Incident \(UTC\):** 11:06 - 12:07 **Service\(s\) Affected:** All APIs **Impact Duration:** 61 minutes ## Summary For 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring. ## Impact on Customers * **APIs:** High latency, or a 500 Internal Server Error. * **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter. * **Geographic Regions Affected \(if applicable\):** 1Password USA/Global ## What Happened? * **Timeline of Events \(UTC\):** * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance. * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates. * 11:19: Automation scal

1. Postmortem · Sep 10, 21:59 UTC
   # Performance Degradation **Date of Incident:** 2025-09-03 **Time of Incident \(UTC\):** 11:06 - 12:07 **Service\(s\) Affected:** All APIs **Impact Duration:** 61 minutes ## Summary For 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring. ## Impact on Customers * **APIs:** High latency, or a 500 Internal Server Error. * **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter. * **Geographic Regions Affected \(if applicable\):** 1Password USA/Global ## What Happened? * **Timeline of Events \(UTC\):** * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance. * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates. * 11:19: Automation scaled up infrastructure to service additional load * 11:30: Increased errors trigger escalation, on-call engineer begins investigation * 11:51: Engineers declare an incident and alert response teams * 12:02: Response team begins restarting affected servers. * 12:07: All servers completed restarts,
2. Resolved · Sep 3, 11:00 UTC
   This incident has been resolved.