Increased Job Queue Times: Windows, Android, and GPU
Aug 4, 19:27 UTCAug 4, 23:40 UTC
Duration
4h 13m
Impact
Minor
Root cause
Dependency
CircleCI, 90 days
28 incidents
Affected
Machine JobsWindows Jobs
Final update
Queue times for Android and Windows jobs have reduced significantly over the last hour. You may still notice brief, isolated queuing at times, but this is no longer at incident-level impact. This was related to capacity constraints with a third-party infrastructure provider. We thank you for your patience while we monitored the situation. If you have any issues, please reach out to our Support team.
Timeline
- Resolved · Aug 4, 23:40 UTC
Queue times for Android and Windows jobs have reduced significantly over the last hour. You may still notice brief, isolated queuing at times, but this is no longer at incident-level impact. This was related to capacity constraints with a third-party infrastructure provider. We thank you for your patience while we monitored the situation. If you have any issues, please reach out to our Support team.
- Monitoring · Aug 4, 22:20 UTC
Queue times for both Android and Windows jobs, affected by capacity constraints with our third-party infrastructure provider, have been recovering significantly for the last half hour. We're continuing to monitor for a bit longer to confirm they return to normal levels. We appreciate your patience.
- Monitoring · Aug 4, 21:46 UTC
Queue times for both Android and Windows jobs continue to be volatile, with sharp swings up and down. This remains related to capacity constraints with our third-party infrastructure provider. Our team will continue to monitor and provide updates at least every 30 minutes. We appreciate your patience.
- Monitoring · Aug 4, 21:16 UTC
Queue times for both Android and Windows jobs, which had been recovering, have started to rise again. This continues to be related to capacity constraints with our third-party infrastructure provider. We'll continue to monitor and provide updates at least every 30 minutes. We appreciate your patience.
- Monitoring · Aug 4, 20:33 UTC
Queue times continue to gradually recover with our third-party infrastructure provider, though we've seen a slight uptick in the last 30 mins. We're keeping an eye on this and will keep you updated as things get back to normal. Again, thank you for your patience.
- Monitoring · Aug 4, 19:59 UTC
We're starting to see queue times slowly recovering. We'll be monitoring this situation, we appreciate your patience.
- Investigating · Aug 4, 19:27 UTC
We are experiencing increased queue times for Windows, Android, and GPU jobs due to capacity constraints with a third-party infrastructure provider. Jobs will continue to be processed but may take longer than usual to start. Thank you for your patience while our engineers work to resolve this.
More from CircleCI
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:13 UTC | Delays in processing new signups and plan changes | none | 1h 53m |
| Sep 2116:22 UTC | Delays starting Linux Machine jobs | minor | 47m |
| Sep 1614:55 UTC | Customers may experience delays in UI updates | minor | 4h 35m |
| Sep 1514:42 UTC | Customers may experience delays in UI updates | minor | 2h 20m |
| Sep 1411:59 UTC | Increased wait times for macOS jobs | minor | 1h |
| Sep 1020:25 UTC | Issues loading pipelines for a small number of projects | minor | 1m |
Also caused by third-party dependency
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1609:48 UTC | Support Portal Service Disruption | minor | 11h 21m | |
| Sep 1608:31 UTC | Support Helpdesk Availability Issues | minor | 6h 27m | |
| Sep 417:03 UTC | Degradation of Hosted Grafana in US Central Region | major | 3h 39m | |
| Sep 416:05 UTC | Elevated Linux worker queue times | major | 15h 4m | |
| Sep 115:25 UTC | Investigating issues in US Central (prod-us-central-0, prod-us-central-5) | minor | 4h 54m | |
| Aug 3106:26 UTC | Packet loss in ORD | minor | 1h 44m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.