Jobs stuck in running state

Final update

Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC. What was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window. Resolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them. The incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures

1. Resolved · Dec 3, 18:12 UTC
   Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC. What was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window. Resolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them. The incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures