EAS Outage

Final update

This incident has been resolved.

1. Resolved · Jun 1, 03:47 UTC
   This incident has been resolved.
2. Monitoring · Jun 1, 03:46 UTC
   Our data center confirmed the network is fully operational and we are closing this incident after confirming EAS iOS build and submission jobs and macOS CI/CD workflow jobs are running as normal. All paid customer jobs have been processed and new paid, high-priority jobs will run with approximately zero queue time. There is a backlog of free jobs we estimate will take approximately 40 minutes to process. Thank you again for your understanding and patience. Building developer trust is our guiding principle at Expo and we work continuously at this. We will internally discuss how to make an incident like this one go more smoothly if were to happen again.
3. Monitoring · Jun 1, 01:37 UTC
   The data center’s network has partially recovered and the availability of the affected services (iOS build and submission jobs and macOS CI jobs) is back. We are monitoring service health. The backlog of affected jobs is being processed as fast as possible - specifically, compared to before this incident about twice as many jobs are running. Thank you for your patience and understanding.
4. Identified · Jun 1, 01:02 UTC
   Our data center provider is still working on the issue. We noticed one of our racks became reachable again and started processing jobs successfully but that rack is now offline again. We are monitoring the servers’ network and status updates from the data center and are ready to work on restoring availability of iOS build jobs and macOS workflows when we get confirmation the data center is healthy again. Thank you for your understanding.
5. Identified · May 31, 23:43 UTC
   There is a networking outage affecting our macOS data center provider. They are working in coordination with their upstream providers to resolve the issue. We will be ready to work immediately on restoring availability once the data center provider is back online.
6. Identified · May 31, 20:38 UTC
   There is a networking outage in our macOS data center. We are waiting for a fix on their end.