High number of 5XX on the Machines API and dashboard
Jul 20, 07:10 UTCJul 20, 17:09 UTC
Duration
9h 59m
Impact
Critical
Root cause
Not disclosed
Fly.io, 90 days
51 incidents
Affected
DashboardMachines API
Final update
This incident has been resolved.
Timeline
- Resolved · Jul 20, 17:09 UTC
This incident has been resolved.
- Monitoring · Jul 20, 13:34 UTC
We are still working on fixing degraded Managed Postgres clusters.
- Monitoring · Jul 20, 09:14 UTC
Some Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected.
- Monitoring · Jul 20, 08:02 UTC
A fix has been implemented and we are monitoring the results.
- Identified · Jul 20, 07:52 UTC
We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.
- Identified · Jul 20, 07:51 UTC
We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.
- Investigating · Jul 20, 07:48 UTC
We are continuing to investigate this issue.
- Investigating · Jul 20, 07:10 UTC
Existing machines are unaffected. We are investigating the issue.
More from Fly.io
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:44 UTC | Partial Sprites outage | major | 7h 28m |
| Sep 2314:47 UTC | Elevated private networking errors | major | 36m |
| Sep 2312:42 UTC | Upstash unavailability in FRA region | major | 1h 22m |
| Sep 2306:18 UTC | IPv6 Networking Issues in DFW | major | 5h 57m |
| Sep 1504:08 UTC | Depot builder failures | minor | 1h 25m |
| Sep 1221:22 UTC | Network issues in US West Coast | minor | 50m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.