Mac jobs failing to start

Final update

Mac jobs have now fully recovered. Thank you for your patience during this incident.

1. Resolved · Jun 1, 00:45 UTC
   Mac jobs have now fully recovered. Thank you for your patience during this incident.
2. Monitoring · Jun 1, 00:42 UTC
   We have processed the backlog of Mac jobs and job start times are recovering to normal parameters.
3. Monitoring · Jun 1, 00:24 UTC
   We are seeing Mac jobs starting successfully and improved network connectivity to the Mac fleet. We will continue to monitor the recovery.
4. Identified · May 31, 22:38 UTC
   Mac jobs are continuing to experience failures. Our infrastructure provider is reporting they are experiencing a DDoS attack causing high network latency and packet loss for our Mac fleet. We are working to bring up additional capacity to restore service. We will continue to update as more information is available.
5. Identified · May 31, 21:24 UTC
   We are continuing to investigate failures with Mac jobs starting due to high network latency and packet loss. Our engineers remain engaged with our infrastructure provider to restore service. We will provide another update within the hour, or sooner if circumstances change. We thank you for your patience while our engineers work to resolve this.
6. Identified · May 31, 20:05 UTC
   We are still seeing failures with Mac jobs starting due to high network latency and packet loss. We will continue to update as we have more information on a resolution.
7. Identified · May 31, 19:09 UTC
   We are continuing to work on resolving the networking issue for Mac jobs.
8. Identified · May 31, 18:15 UTC
   We have identified a network outage that is preventing Mac jobs from starting. This include m4pro.medium and m4pro.large resource types. We are working with our infrastructure provider to restore service. We will update this incident as we work to resolve the issue.