Partial outage in ORD

Final update

This incident has been resolved.

1. Resolved · Jul 3, 05:59 UTC
   This incident has been resolved.
2. Monitoring · Jul 3, 04:41 UTC
   Customer workloads are now starting and we're monitoring the affected hosts. Affected Managed Postgres instances will be investigated.
3. Identified · Jul 3, 04:21 UTC
   Power restoration is ongoing and we're making sure the hosts are healthy before starting customer workloads to avoid issues. Customer impact remains and updates to come.
4. Identified · Jul 3, 02:59 UTC
   Power restoration work in a subset of ORD is still in progress and impact remains ongoing for a subset of hosts and some Managed Postgres clusters.
5. Identified · Jul 3, 00:54 UTC
   Our provider has advised us their facilities team is working on restoring the power, we'll provide another update as soon as we learn more.
6. Identified · Jul 3, 00:29 UTC
   We've identified and reported power issues with one of our upstream providers in ORD. We're waiting for an update from our upstream for a resolution. Some Managed Postgres clusters in ORD will be unavailable due to placement.
7. Investigating · Jul 3, 00:11 UTC
   We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time. Some Managed Postgres clusters in ORD region may be unavailable at this time.