Some users are unable to interact with the admin console

Final update

# Incident Postmortem - Some customers are unable to interact with the admin console **Date of Incident:** 2025-09-24 **Time of Incident \(UTC\):** 02:27 - 17:16 **Service\(s\) Affected:** Admin console, Sign in **Impact Duration:** 36:49 ## Summary Some customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console. ## Impact on Customers * **Admin console:** Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages. * **Log in:** Affected customers were also unable to log in to the application. * **Number of Affected Customers \(approximate\):** 515 * **Geographic Regions Affected \(if applicable\):** All regions ## What Happened? * **Timeline of Events \(UTC\):** * Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking * Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag. * Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day.

1. Postmortem · Oct 8, 20:45 UTC
   # Incident Postmortem - Some customers are unable to interact with the admin console **Date of Incident:** 2025-09-24 **Time of Incident \(UTC\):** 02:27 - 17:16 **Service\(s\) Affected:** Admin console, Sign in **Impact Duration:** 36:49 ## Summary Some customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console. ## Impact on Customers * **Admin console:** Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages. * **Log in:** Affected customers were also unable to log in to the application. * **Number of Affected Customers \(approximate\):** 515 * **Geographic Regions Affected \(if applicable\):** All regions ## What Happened? * **Timeline of Events \(UTC\):** * Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking * Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag. * Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day. * Sep 24th 10:03pm: Merged an application change to revert the change to prevent the issue reoccurring. * Sep 25th 5:03pm: The change was deployed with scheduled application release, error rate dropped off shortly after. * **Root Cause Analysis:** The issue was caused by GET requests to the Users
2. Resolved · Sep 24, 04:10 UTC
   This incident has been resolved.
3. Monitoring · Sep 24, 03:57 UTC
   Our engineering team has rolled out mitigation to remedy affected users. We are monitoring the results.
4. Identified · Sep 24, 03:44 UTC
   Our engineering team has identified the issue and are working towards mitigating.
5. Investigating · Sep 24, 03:36 UTC
   We are continuing to investigate the issue.
6. Investigating · Sep 24, 03:00 UTC
   We are actively investigating an issue where some users are encountering errors when interacting with the admin console, which leads to errors during sign-in.