Latency issues across a number of services

Final update

On July 23, 2026, between 07:08 and 09:39 UTC, several services experienced delays: 8% of actions workflow runs experienced an average run start delay of 10 minutes, 5% of webhook deliveries exceeded SLO, and code scanning, repos, notifications, issues and pull requests experienced increased latency over the life of the incident. The root cause of the incident was a node of our background job processing system which did not recover after entering scheduled host maintenance. The incident was mitigated by identifying the problematic shard and restoring its correct state, after which queue backlogs drained and services recovered. To speed mitigation, we have added monitors for nodes in this unhealthy state after maintenance operations. To prevent future recurrence, we are adapting our lifecycle automation to verify host rejoin after a scheduled reboot.

1. Resolved · Jul 23, 09:39 UTC
   On July 23, 2026, between 07:08 and 09:39 UTC, several services experienced delays: 8% of actions workflow runs experienced an average run start delay of 10 minutes, 5% of webhook deliveries exceeded SLO, and code scanning, repos, notifications, issues and pull requests experienced increased latency over the life of the incident. The root cause of the incident was a node of our background job processing system which did not recover after entering scheduled host maintenance. The incident was mitigated by identifying the problematic shard and restoring its correct state, after which queue backlogs drained and services recovered. To speed mitigation, we have added monitors for nodes in this unhealthy state after maintenance operations. To prevent future recurrence, we are adapting our lifecycle automation to verify host rejoin after a scheduled reboot.
2. Monitoring · Jul 23, 09:39 UTC
   The degradation has been mitigated. We are monitoring to ensure stability.
3. Investigating · Jul 23, 09:35 UTC
   Webhooks is operating normally.
4. Investigating · Jul 23, 09:27 UTC
   The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.
5. Investigating · Jul 23, 09:22 UTC
   We identified the source of latency affecting multiple services and applied a fix. Issues and Actions are recovering, and remaining affected services are seeing improvement as processing backlogs clear. We are actively monitoring recovery across all services.
6. Investigating · Jul 23, 09:19 UTC
   The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
7. Investigating · Jul 23, 09:18 UTC
   The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.
8. Investigating · Jul 23, 08:34 UTC
   We're currently investigating latency across multiple services. This can show as Actions jobs taking longer to start, Issues search serving stale results, and other listed services being similarly impacted.
9. Investigating · Jul 23, 08:25 UTC
   Pull Requests is experiencing degraded performance. We are continuing to investigate.
10. Investigating · Jul 23, 07:53 UTC
    We are investigating reports of degraded availability for Actions, Issues and Webhooks