Skip to content
ClickHouse · Data and observabilitySep 15, 2026, 18:57 UTC

ClickPipes failing on Kinesis in AWS us-east-1

CriticalBugUpdated 16h ago
Sep 15, 18:57 UTCSep 17, 00:43 UTC
Duration
29h 46m
Impact
Critical
Root cause
Bug
ClickHouse, 90 days
16 incidents
Affected
Not listed by the vendor.
Status page

Final update

This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues:

Timeline

  1. Resolved · Sep 17, 00:43 UTC
    This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues:
  2. Identified · Sep 16, 15:21 UTC
    AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:::stream//consumer/`) rather than the stream resource (for example, `arn:aws:kinesis:::stream/`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException. AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
  3. Identified · Sep 16, 03:17 UTC
    AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:::stream//consumer/`) rather than the stream resource (for example, `arn:aws:kinesis:::stream/`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException. AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.
  4. Identified · Sep 15, 22:45 UTC
    AWS has confirmed the issue on their end and are in the process of rolling back the change. The ETA for this rollback in us-east-1 is by end of day tomorrow. Signs of gradual recovery are expected in us-east-1 as the rollback progresses across AZs during this time. In the meantime, if your ClickPipe is still affected, please update your IAM policy as instructed in the previous status update.
  5. Identified · Sep 15, 20:03 UTC
    The issue has been identified as a change on AWS related to Kinesis IAM permissions and appears to be currently isolated to us-east-1. While we work with AWS to mitigate this issue, we recommend to update your IAM policy to include a trailing wildcard (`/*`) to the resource for the `kinesis:RegisterStreamConsumer` action like so: ``` { "Effect": "Allow", "Action": [ "kinesis:RegisterStreamConsumer" ], "Resource": [ "arn:aws:kinesis:{REGION}:{ACCOUNT_ID}:stream/{STREAM_NAME}/*" ] } ``` Once this is applied, you must restart your ClickPipe for the changes to take affect.
  6. Investigating · Sep 15, 18:57 UTC
    Currently investigating failures on AWS us-east-1 for customers using ClickPipes on Kinesis in this region.

More from ClickHouse

Full history

Also caused by software bug

All
StartedIncidentDuration
Sep 318:20 UTCRetroactive Incident: Twilio Personalized Support Phone Line AffectedTwilio0m
Aug 819:48 UTCAlerting expressions pipeline failing when recovery settingsGrafana Labs0m
Aug 615:22 UTCIncident with ActionsGitHub10h 42m
Jul 2314:14 UTC[Medium] Issues with Box HubsBox16m
Jun 1719:00 UTCIncident With WebhooksGitHub0m
May 2801:13 UTCWebhook APIs and UI DegradedGitHub19m

From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.

Weekly: the week's major outages, postmortems and breaches, Saturday mornings.