ClickPipes failing on Kinesis in AWS us-east-1
Sep 15, 18:57 UTCSep 17, 00:43 UTC
Duration
29h 46m
Impact
Critical
Root cause
Bug
ClickHouse, 90 days
16 incidents
Affected
Not listed by the vendor.
Final update
This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues:
Timeline
- Resolved · Sep 17, 00:43 UTC
This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues:
- Identified · Sep 16, 15:21 UTC
AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:::stream//consumer/`) rather than the stream resource (for example, `arn:aws:kinesis:::stream/`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException. AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
- Identified · Sep 16, 03:17 UTC
AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:::stream//consumer/`) rather than the stream resource (for example, `arn:aws:kinesis:::stream/`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException. AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
- Identified · Sep 15, 22:45 UTC
AWS has confirmed the issue on their end and are in the process of rolling back the change. The ETA for this rollback in us-east-1 is by end of day tomorrow. Signs of gradual recovery are expected in us-east-1 as the rollback progresses across AZs during this time. In the meantime, if your ClickPipe is still affected, please update your IAM policy as instructed in the previous status update.
- Identified · Sep 15, 20:03 UTC
The issue has been identified as a change on AWS related to Kinesis IAM permissions and appears to be currently isolated to us-east-1. While we work with AWS to mitigate this issue, we recommend to update your IAM policy to include a trailing wildcard (`/*`) to the resource for the `kinesis:RegisterStreamConsumer` action like so: ``` { "Effect": "Allow", "Action": [ "kinesis:RegisterStreamConsumer" ], "Resource": [ "arn:aws:kinesis:{REGION}:{ACCOUNT_ID}:stream/{STREAM_NAME}/*" ] } ``` Once this is applied, you must restart your ClickPipe for the changes to take affect.
- Investigating · Sep 15, 18:57 UTC
Currently investigating failures on AWS us-east-1 for customers using ClickPipes on Kinesis in this region.
More from ClickHouse
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2315:28 UTC | We are investigating an issue affecting Azure eastus2 region | minor | 30m |
| Sep 2213:19 UTC | We are investigating an issue with CH servers in Azure germanywestcentral region | minor | 5h 6m |
| Sep 1610:16 UTC | Intermittent Issues with the Support tab | minor | 13h 50m |
| Sep 1420:36 UTC | Distributed Cache performance degradation in AWS us-east-1 | minor | 3h 59m |
| Sep 116:11 UTC | Service degradation in GCP us-central-1 | none | 19m |
| Aug 2310:14 UTC | Slow provisioning times of services in GCP europe-west4 region | minor | 32m |
Also caused by software bug
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 318:20 UTC | Retroactive Incident: Twilio Personalized Support Phone Line Affected | none | 0m | |
| Aug 819:48 UTC | Alerting expressions pipeline failing when recovery settings | minor | 0m | |
| Aug 615:22 UTC | Incident with Actions | critical | 10h 42m | |
| Jul 2314:14 UTC | [Medium] Issues with Box Hubs | major | 16m | |
| Jun 1719:00 UTC | Incident With Webhooks | none | 0m | |
| May 2801:13 UTC | Webhook APIs and UI Degraded | minor | 19m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.