Disruption with some GitHub services

Final update

From January 9 13:11 UTC to January 12 10:17 UTC, new Linux Custom Images generated for Larger Hosted Runners were broken and not able to run jobs. Customers who did not generate new Custom Images during this period were not impacted. This issue was caused by a change to improve reliability of the image creation process. Due to a bug, the change triggered an unrelated protection mechanism which determines if setup has already been attempted on the VM and caused the VM to be marked unhealthy. Only Linux images which were generated while the change was enabled were impacted. The issue was mitigated by rolling back the change. We are improving our testing around Custom Image generation as part of our GA readiness process for the public preview feature.. This includes expanding our canary suite to detect this and similar interactions as part of a controlled rollout in staging prior to any customer impact.

1. Resolved · Jan 12, 10:17 UTC
   From January 9 13:11 UTC to January 12 10:17 UTC, new Linux Custom Images generated for Larger Hosted Runners were broken and not able to run jobs. Customers who did not generate new Custom Images during this period were not impacted. This issue was caused by a change to improve reliability of the image creation process. Due to a bug, the change triggered an unrelated protection mechanism which determines if setup has already been attempted on the VM and caused the VM to be marked unhealthy. Only Linux images which were generated while the change was enabled were impacted. The issue was mitigated by rolling back the change. We are improving our testing around Custom Image generation as part of our GA readiness process for the public preview feature.. This includes expanding our canary suite to detect this and similar interactions as part of a controlled rollout in staging prior to any customer impact.