Inaccurate Estimated Billing Data

Final update

Between July 16 at 7:38 PM PDT and July 18 at 6:00 AM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console, including the Cost and Usage Report. Customers may have received erroneous budget and cost anomaly detection alerts, and observed inflated estimated cost and usage data. On July 16 at 7:46 PM PDT, our alarms detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams. We were alerted to this issue on July 17 at 12:19 AM PDT by customer escalations, and immediately began to investigate. We first informed customers via AWS Health on July 17 at 1:33 AM. At 8:24 AM PDT we paused further updates to estimated billing data and turned off budget and cost anomaly alerts as a precautionary measure. We identified the root cause on July 17 at 12:00 PM PDT as a configuration change in our bill computation system. This system relies on unit conversion data to calculate line item charges. The configuration change caused updates to the unit conversion data to fail, resulting in inflated line item costs, which propagated to the Billing and Cost Management console and triggered budget and cost a

1. Resolved · Jul 18, 13:57 UTC
   Between July 16 at 7:38 PM PDT and July 18 at 6:00 AM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console, including the Cost and Usage Report. Customers may have received erroneous budget and cost anomaly detection alerts, and observed inflated estimated cost and usage data. On July 16 at 7:46 PM PDT, our alarms detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams. We were alerted to this issue on July 17 at 12:19 AM PDT by customer escalations, and immediately began to investigate. We first informed customers via AWS Health on July 17 at 1:33 AM. At 8:24 AM PDT we paused further updates to estimated billing data and turned off budget and cost anomaly alerts as a precautionary measure. We identified the root cause on July 17 at 12:00 PM PDT as a configuration change in our bill computation system. This system relies on unit conversion data to calculate line item charges. The configuration change caused updates to the unit conversion data to fail, resulting in inflated line item costs, which propagated to the Billing and Cost Management console and triggered budget and cost anomaly alerts. We mitigated the issue on July 17 at 12:30 PM PDT which corrected the unit conversion configuration, and began reprocessing cost and usage data for all customer accounts. We started observing recovery at 4:19 PM PDT, and the majority of accounts were fully recovered by July 18 at 6:00
2. Update · Jul 18, 08:05 UTC
   We continue to make substantial progress toward resolving the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. Our mitigation efforts are working as expected and we are seeing an increasing number of accounts reflecting correct cost and usage data. We expect all affected accounts to be fully recovered by July 19, 12:00 AM PDT. Until the backfill is complete, some customers may still observe incorrect cost and usage data in the Billing and Cost Management Console and Cost and Usage Reports. These estimates do not reflect actual usage or charges. We will provide another update by July 18, 7:00 AM PDT, or sooner if additional information becomes available.
3. Update · Jul 18, 01:38 UTC
   We continue to make steady progress toward resolving the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. Our efforts to backfill corrected data remain underway, and we expect all affected accounts to be fully recovered by July 19, 12:00 AM PDT. Until the backfill is complete, some customers may still observe incorrect cost and usage data in the Billing and Cost Management Console and Cost and Usage Reports. These estimates do not reflect actual usage or charges. Customers who configured their Cost and Usage Report with the "Overwrite" option require no action , their report will be automatically updated with corrected data once the backfill completes. Customers who configured their Cost and Usage Report with the "Create new report versions" option retain all previous report deliveries in their S3 bucket. The report version delivered during the impacted window may contain inaccurate data. Once the data backfill is complete, a corrected report version will be delivered under a new assemblyId. Customers using this configuration should update any downstream processes (Athena tables, Redshift pipelines, Amazon QuickSight, or custom ETL) to reference the latest assemblyId for the affected billing period, and may delete or archive the impacted report version to prevent processing stale data. To identify the latest report, customers can follow the steps in our documentation. We will provide another update by July 18, 1:00 AM PDT, o
4. Update · Jul 17, 19:56 UTC
   Our efforts to backfill corrected estimated cost and usage data are still underway. We are progressing slower than anticipated. While we are seeing some accounts recover with correct cost and usage data, we expect all affected accounts to be recovered by July 19 12:00 AM PDT. Until the backfill is complete, some customers may still see incorrect cost and usage data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 7:00 PM, or sooner if information becomes available.
5. Update · Jul 17, 16:59 UTC
   We have identified the root cause and mitigated the underlying issue causing incorrect estimated cost and usage data to be displayed in the Billing and Cost Management Console, and Cost and Usage Reports. We have begun backfilling data to correct cost data for all customers. We expect some customers to begin seeing recovery within the next three hours, and full recovery for all customers by July 18 12:00 PM PDT. Until the backfill is complete, some customers may still see incorrect cost and usage data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 1:00 PM, or sooner if information becomes available.
6. Update · Jul 17, 14:53 UTC
   We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console, including the Cost and Usage Report. The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused. We are in the process of reverting to the last accurate estimated billing data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We expect this mitigation to take several hours to complete as we work through recomputing the estimated billing data. We will provide another update by 10:00 AM PDT or sooner if more information becomes available.
7. Update · Jul 17, 12:54 UTC
   We continue to work on multiple mitigation paths in parallel to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console, including the Cost and Usage Report. We are evaluating resuming estimated billing computations, as our internal monitoring indicates the billing computation subsystem is now producing accurate estimates. We are conducting additional validation before proceeding with this path. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 8:00 AM PDT or sooner if more information becomes available.
8. Update · Jul 17, 11:58 UTC
   We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We are actively working on multiple mitigation paths in parallel. The first path involves reverting to the last known good estimated bill computation. With this approach, customers will only see cost and usage data through July 15, however the inflated cost data will be removed. The second path involves rolling back a recent change to the billing computation subsystem. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 6:00 AM PDT or sooner if more information becomes available.
9. Update · Jul 17, 10:52 UTC
   We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. As previously shared, we have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem. To prevent further inaccurate billing estimates from being displayed, we have paused estimated billing computations. Customers who are currently seeing normal bill estimates will continue to see those estimates, and customers who are seeing inflated estimates will not see them increase further while we work toward resolution. The displayed billing estimates do not reflect actual usage and charges. We continue to work on fully mitigating the issue. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. There are no customer actions required at this time. We will provide another update by 5:00 AM PDT or sooner if more information becomes available.
10. Update · Jul 17, 10:03 UTC
    We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.
11. Update · Jul 17, 09:07 UTC
    Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
12. Update · Jul 17, 08:33 UTC
    We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.