Confluent Cloud Console Degraded (All Regions) and Multiple Services Impacted in AWS us-west-2

Final update

This incident has been resolved. All affected services, including Flink in AWS us-west-2, have fully recovered, and no further customer impact is expected. No action is required from customers.

1. Resolved · Sep 22, 01:31 UTC
   This incident has been resolved. All affected services, including Flink in AWS us-west-2, have fully recovered, and no further customer impact is expected. No action is required from customers.
2. Monitoring · Sep 22, 00:40 UTC
   All affected services, including Flink in AWS us-west-2, have recovered and we are no longer observing customer impact. We are monitoring to confirm full stability before resolving this incident.
3. Identified · Sep 22, 00:06 UTC
   Confluent is continuing to monitor recovered services and complete recovery for a small number of Flink customers in AWS us-west-2 that may still be experiencing degraded jobs. We are actively working to fully restore them and will post the next update by 00:30 UTC, or sooner if service is fully restored.
4. Identified · Sep 21, 23:35 UTC
   All affected services have recovered with the exception of a small number of Flink customers in AWS us-west-2 that may still experience degraded jobs while we complete recovery. We are actively working to fully restore them and will post the next update by 00:00 UTC, or sooner if service is fully restored.
5. Identified · Sep 21, 23:06 UTC
   Flink in AWS us-west-2 is still recovering as we continue to restore full service: while some jobs may start to recover, others may remain degraded and new job submissions may still fail. Previously affected services have recovered and are being monitored for stability. We will post the next update by 23:30 UTC or sooner if recovery completes sooner.
6. Identified · Sep 21, 22:12 UTC
   Flink in AWS us-west-2 is still recovering as we continue to restore full service: running jobs may remain degraded and new job submissions may still fail. Previously affected services have recovered and are being monitored for stability. We will post the next update by 22:30 UTC or sooner if recovery completes sooner.
7. Identified · Sep 21, 21:30 UTC
   Most affected services are recovering and we are continuing to monitor for stability. Flink in AWS us-west-2 is still recovering and running jobs may remain degraded with new submissions still failing. We will post the next update by 22:00 UTC or sooner if the situation changes.
8. Identified · Sep 21, 20:56 UTC
   We have applied a mitigation and most affected services are beginning to recover. Flink in AWS us-west-2 will continue to be affected while we restore full service: running jobs may remain degraded and new job submissions may continue to fail during this period. We will post the next update by 21:30 UTC, or sooner if service is restored.
9. Investigating · Sep 21, 20:03 UTC
   We are continuing to work to resolve the incident and do not yet have a confirmed mitigation. Impact remains as previously described. We will post the next update by 20:30 UTC, or sooner if the situation changes.
10. Investigating · Sep 21, 19:33 UTC
    We are continuing to validate several proposed mitigations to address the ongoing incident affecting the Confluent Cloud UI and other services in the AWS us-west-2 region. We do not yet have a confirmed fix in place. We will post the next update by 20:00 UTC, or sooner if service is restored.
11. Investigating · Sep 21, 19:08 UTC
    We are continuing to investigate an incident affecting the Confluent Cloud UI and other services in the AWS us-west-2 region. Several potential mitigations remain under active testing and validation. We do not yet have a confirmed fix in place. We will post the next update by 19:30 UTC, or sooner if service is restored.
12. Investigating · Sep 21, 18:31 UTC
    We are testing and validating several potential mitigations in parallel to restore the Confluent Cloud Console and affected services. We do not yet have a confirmed fix in place and are continuing to investigate. We will post the next update by 19:00 UTC, or sooner if service is restored.
13. Investigating · Sep 21, 18:01 UTC
    We have refined the scope of customer impact and continue to work on a mitigation. Impacted Services: - Confluent Cloud Console (web UI), all regions: broad degradation. Customers may be unable to view or manage their resources in the Console. - Schema Registry, AWS us-west-2: a subset of Schema Registry clusters are unavailable. - Flink, AWS us-west-2: all running jobs are degraded, and new job submissions will fail. - Cluster Linking, AWS us-west-2: links to Enterprise clusters are impacted. - Topic operations, AWS us-west-2: unavailable for Freight and some Basic and Standard clusters. - Tableflow, AWS us-west-2: All operations are unavailable (additionally, topics utilizing the us-west-2 Schema Registry may be impacted) - Connect, AWS us-west-2: Some connectors dependent on unavailable Schema Registry clusters may be affected - Notifications, all regions: delivery is delayed. Customers may not receive notifications and alerts until the backlog clears. Confirmed not impacted: - Regular produce and consume operations on Kafka clusters are not affected. - Topic-related operations on Enterprise and Dedicated clusters are not affected. We are actively applying a mitigation. We will post the next update by 18:30 UTC, or sooner if service is restored.
14. Investigating · Sep 21, 17:04 UTC
    We are confirming a contributing factor and applying a mitigation intended to restore Confluent Cloud services. We will post an update by 17:45 UTC, or sooner if service is restored.
15. Investigating · Sep 21, 16:32 UTC
    We have made progress in identifying the cause of the disruption in AWS us-west-2 and are working to confirm the safest path to mitigation. Some customers in this region may continue to experience errors or degraded performance. We will post the next update by 17:00 UTC, or sooner if mitigation is achieved.
16. Investigating · Sep 21, 16:15 UTC
    Multiple Confluent Cloud services in the AWS us-west-2 region are experiencing elevated error rates and service disruption. Impacted customers may see failed or slow requests to Kafka clusters, Connect, and Flink in this region. The Confluent Cloud UI is also degraded and visibility to clusters outside of us-west-2 may be impacted as well. We are actively investigating and will post the next update by 16:30 UTC, or sooner if we have new information.