Issues with network routing for Mac jobs

Final update

## Summary From 18:00 UTC on July 13, 2026 to 23:18 UTC on July 14, 2026, some customers running jobs on CircleCI's macOS fleet experienced jobs that hung or failed to progress during GitHub fetch steps. For these customers, GitHub fetch steps which would ordinarily take 1-2 minutes took 20-30 minutes, and timed out. The issue was caused by a capacity and routing problem on the network path between our Mac infrastructure provider and GitHub, and the associated intermittent networking issues slowed down fetches from GitHub. During the entire incident, our team worked directly with our Mac infrastructure provider to identify and reroute traffic away from the affected network paths. We resolved an initial occurrence at 19:40 UTC on July 13. The issue recurred at 21:52 UTC that same day, we reopened the incident, and it was fully resolved by 23:18 UTC on July 14. Our macOS fleet had a similar issue several weeks ago. From 20:18 UTC on June 24, 2026 to 03:09 UTC on June 25, 2026, customers experienced a [similar incident](https://status.circleci.com/incidents/gvysjmkf4ct9) affecting the ability of macOS jobs to reach GitHub. That incident was also caused by a network routing problem in

1. Postmortem · Jul 16, 01:56 UTC
   ## Summary From 18:00 UTC on July 13, 2026 to 23:18 UTC on July 14, 2026, some customers running jobs on CircleCI's macOS fleet experienced jobs that hung or failed to progress during GitHub fetch steps. For these customers, GitHub fetch steps which would ordinarily take 1-2 minutes took 20-30 minutes, and timed out. The issue was caused by a capacity and routing problem on the network path between our Mac infrastructure provider and GitHub, and the associated intermittent networking issues slowed down fetches from GitHub. During the entire incident, our team worked directly with our Mac infrastructure provider to identify and reroute traffic away from the affected network paths. We resolved an initial occurrence at 19:40 UTC on July 13. The issue recurred at 21:52 UTC that same day, we reopened the incident, and it was fully resolved by 23:18 UTC on July 14. Our macOS fleet had a similar issue several weeks ago. From 20:18 UTC on June 24, 2026 to 03:09 UTC on June 25, 2026, customers experienced a [similar incident](https://status.circleci.com/incidents/gvysjmkf4ct9) affecting the ability of macOS jobs to reach GitHub. That incident was also caused by a network routing problem in our Mac infrastructure provider. We thank our customers for their patience while we worked through this incident. Please see below for specific actions which CircleCI and our Mac infrastructure provider will be taking. Please reach out to our support team with any questions or concerns. The status p
2. Resolved · Jul 14, 23:18 UTC
   We continue to see significant VCS fetch time recovery over the last couple of hours. We're going to resolve this incident, and our engineering team will continue to monitor closely. We greatly appreciate your patience as we worked through this with our Mac infrastructure provider. Please reach out to our Support team if you have any issues.
3. Monitoring · Jul 14, 22:15 UTC
   Our Mac infrastructure provider has shifted more traffic to working network providers and we're starting to see significant recovery again in VCS fetch times. As always, our engineering team will continue to monitor this closely and we'll provide another update soon.
4. Investigating · Jul 14, 20:18 UTC
   We're starting to see VCS fetch times rising again. Our engineering team is actively investigating and working with our Mac infrastructure provider to resolve this. Again, we appreciate your patience.
5. Monitoring · Jul 14, 18:28 UTC
   We're continuing to see slow, progressive recovery in VCS fetch times. We're going to continuing monitoring and update again soon. We appreciate your patience.
6. Monitoring · Jul 14, 16:58 UTC
   Our Mac infrastructure provider has shifted the inbound routes and we are seeing recovery in VCS fetch times. We will be continuing to monitor this issue over the next few hours. We'll update again soon.
7. Identified · Jul 14, 16:31 UTC
   We've identified the issue as some inbound routes from the affected VCS provider causing significantly lower available bandwidth which is negatively impacting repo fetch times. We're working with our Mac infrastructure provider to attempt to shift inbound traffic to a different route, however this process may take an extended period of time to complete. We will continue to provide updates as they become available. As we've shared in previous updates, increasing the no_output_timeout to 15-20 min continues to help: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit
8. Identified · Jul 14, 15:35 UTC
   Our Mac infrastructure provider is testing changes to their network configuration in an effort to isolate the source of the issue. We're actively validating whether this resolves the impact and will follow up with another update soon. In the meantime, increasing the no_output_timeout to 15-20 min continues to help: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit
9. Identified · Jul 14, 13:59 UTC
   Unfortunately there are still some customers that may be experiencing jobs that hang or fail to progress during Github fetch steps due to ongoing intermittent networking issues with our Mac infrastructure provider. Our engineers are actively working with them to resolve this. We appreciate your patience. In the meantime, we've seen success with increasing the no_output_timeout to 15 - 20 min to allow more time for Github fetches. This can be done by following this community guide: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit
10. Monitoring · Jul 13, 23:35 UTC
    We've identified the issue. Some customers may experience jobs that hang or fail to progress during Github fetch steps due to intermittent networking issues. Our engineers are actively monitoring. In the meantime, we've seen success with increasing the no_output_timeout to 15 - 20 min to allow more time for Github fetches. This can be done by following this community guide: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit
11. Investigating · Jul 13, 23:13 UTC
    We've been able to reproduce the issue, but it still appears to be very much intermittent. We're continuing to work with our Mac infrastructure provider to help diagnose the issue. We'll update again soon.
12. Investigating · Jul 13, 22:21 UTC
    We're still investigating these intermittent network issues. We'll update again shortly, we appreciate your patience.
13. Investigating · Jul 13, 21:52 UTC
    We're received some reports of intermittent network issues between our mac infrastructure and a VCS provider again. We are investigating and will provide updates.