Increase in Workflow runs stuck in pending, failed

Final update

The issue affecting workflow runs getting stuck in a pending or failed state has been resolved. New workflow runs after the impact period are unaffected. Affected workflow runs have been recovered where possible. Runs that could be resumed safely were resumed and runs that could not be resumed safely were transitioned to a cancelled state.

1. Resolved · Jul 1, 18:30 UTC
   The issue affecting workflow runs getting stuck in a pending or failed state has been resolved. New workflow runs after the impact period are unaffected. Affected workflow runs have been recovered where possible. Runs that could be resumed safely were resumed and runs that could not be resumed safely were transitioned to a cancelled state.
2. Identified · Jul 1, 17:49 UTC
   We are continuing to work on recovering affected workflow runs that may be stuck in a pending state or have reached a failed state. New workflow runs are not impacted.
3. Identified · Jul 1, 16:57 UTC
   We've identified an issue where some workflow runs created between June 30 at 22:24 UTC and July 1 at 16:08 UTC may be stuck in a pending state or have reached a failed state. New workflow runs are not impacted. We are working on a fix for runs stuck in a pending state and will provide additional updates as they become available.