Snyk recurring tests showing slow performance

Final update

We are no longer observing degradation affecting Recurring Tests. The Snyk Incident Response has now been stood down.

1. Resolved · May 5, 12:23 UTC
   We are no longer observing degradation affecting Recurring Tests. The Snyk Incident Response has now been stood down.
2. Identified · May 1, 10:52 UTC
   Our investigation into the degradation affecting Recurring Tests is ongoing. Some customers may continue to notice that a small percentage of their Recurring Tests are not executing as scheduled.
3. Identified · Apr 30, 08:51 UTC
   We are continuing to observe the degradation affecting our Recurring Tests, and our investigations remain ongoing. Some customers may notice that a small percentage of their Recurring Tests are not executing as scheduled.
4. Identified · Apr 29, 06:31 UTC
   We have observed further degradation and are investigating. Customers may observe a small percentage of Recurring Tests not executing as scheduled.
5. Monitoring · Apr 27, 09:44 UTC
   We are no longer observing delays affecting Recurring Tests. Monitoring is now underway to ensure there are no further delays during today's schedule.
6. Investigating · Apr 24, 09:11 UTC
   We are continuing our mitigation efforts. However, customers will continue to experience delays and, in some cases, failures of their scheduled recurring tests. Further updates will follow.
7. Investigating · Apr 22, 13:02 UTC
   We have identified the root cause of the degraded performance affecting recurring tests and are working towards a fix. Customers will continue to experience delays and, in some cases, failures of their scheduled recurring tests. We will continue to keep you updated with our progress.
8. Investigating · Apr 22, 09:03 UTC
   Our investigations remain ongoing. However, we are still observing degradation affecting our daily and weekly recurring tests. We will continue to keep you updated with our progress.
9. Investigating · Apr 21, 18:58 UTC
   We’ve identified a regression impacting our recurring test performance. Specifically, both daily and weekly test suites are showing increased latency. Users affected by this degradation may see some tests failing intermittently or, in some cases, failing to execute entirely. Our engineers are currently investigating the root cause, and a fix is being prioritized. We will provide another update once the stabilization patch is deployed.