Some customers are unable to access their organizations
Jun 29, 10:23 UTCJun 29, 16:58 UTC
Duration
6h 35m
Impact
Minor
Root cause
Not disclosed
CircleCI, 90 days
27 incidents
Affected
CircleCI UI
Final update
This incident has been resolved.
Timeline
- Resolved · Jun 29, 16:58 UTC
This incident has been resolved.
- Monitoring · Jun 29, 16:44 UTC
Error rates viewing orgs are recovering. We will continue to monitor to verify the fix resolves the issues.
- Identified · Jun 29, 15:54 UTC
We have identified the cause of errors accessing orgs. We are implementing a fix, and will monitor for recovery as the change rolls out.
- Investigating · Jun 29, 12:16 UTC
We are continuing to investigate reports that some customers cannot access their organizations in CircleCI. Our investigation continues to point to an authorization issue between CircleCI and GitHub. Some affected users have received a 403 from GitHub indicating the resource is SAML protected and the OAuth token may not be authorized. We previously shared re-authenticating as a potential workaround. We have since received reports that this didn't restore access for all affected customers, so we don't consider it a reliable resolution at this time. Our engineering team is continuing to investigate this to determine the cause and next steps. We will post another update as soon as we have more information. We apologize for the continued disruption and appreciate your patience.
- Investigating · Jun 29, 11:44 UTC
We are still investigating reports that some customers cannot access their organizations in CircleCI. Early findings suggest the problem may be related to GitHub authorization, where affected users receive a 403 from GitHub indicating the resource is SAML protected and the OAuth token may not be authorized. We have not yet confirmed this as the root cause. As a potential workaround, some users may be able to restore access by performing a re-authentication: https://support.circleci.com/hc/en-us/articles/360051228052-How-to-Perform-a-Re-Authentication We will post another update once we have more certainty.
- Investigating · Jun 29, 11:33 UTC
We are continuing to investigate reports that some customers are unable to access their organizations in CircleCI. Those affected may see errors when loading their organizations. Our engineering team are still working to identify the cause. We will share another update as soon as we have more information.
- Investigating · Jun 29, 10:23 UTC
We are investigating reports that some customers are unable to access their organizations in CircleCI. Those affected may see errors when loading their organizations. Our engineering team is actively investigating.
More from CircleCI
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:13 UTC | Delays in processing new signups and plan changes | none | 1h 53m |
| Sep 2116:22 UTC | Delays starting Linux Machine jobs | minor | 47m |
| Sep 1614:55 UTC | Customers may experience delays in UI updates | minor | 4h 35m |
| Sep 1514:42 UTC | Customers may experience delays in UI updates | minor | 2h 20m |
| Sep 1411:59 UTC | Increased wait times for macOS jobs | minor | 1h |
| Sep 1020:25 UTC | Issues loading pipelines for a small number of projects | minor | 1m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.