Increased Error Rates

Final update

Between 11:27 AM and 12:20 PM PST we experienced substantial error rates for S3 PUT/GET requests in EU-CENTRAL-2 Region. Engineers were engaged immediately based on automated alarming. We identified the root cause as an issue with a subsystem responsible for assembling objects bytes in storage. At 12:04 PM PST, we implemented mitigations and began observing early signs of recovery for S3. Error rates continued to improve, and other AWS Services continued to recover until 12:50 PM PST when we observed full recovery. We continue to work toward backfilling Cloudwatch logs, and expect that to continue over the next couple hours. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

1. Resolved · Mar 7, 21:04 UTC
   Between 11:27 AM and 12:20 PM PST we experienced substantial error rates for S3 PUT/GET requests in EU-CENTRAL-2 Region. Engineers were engaged immediately based on automated alarming. We identified the root cause as an issue with a subsystem responsible for assembling objects bytes in storage. At 12:04 PM PST, we implemented mitigations and began observing early signs of recovery for S3. Error rates continued to improve, and other AWS Services continued to recover until 12:50 PM PST when we observed full recovery. We continue to work toward backfilling Cloudwatch logs, and expect that to continue over the next couple hours. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.
2. Update · Mar 7, 20:28 UTC
   We are seeing early signs of recovery and continue to monitor and work toward full recovery.
3. Update · Mar 7, 20:17 UTC
   We can confirm substantial error rates for PUT and GET requests to Amazon S3 in the EU-CENTRAL-2 Region. Engineers engaged immediately based on automated alarming. We have triangulated the issue to a subsystem responsible for assembling objects from bytes in storage. We have begun implementing mitigations, and are observing some improvement in error rates. We continue to work to identify the root cause, and are working on multiple parallel paths to fully mitigate the issue. Other AWS Services (such as EC2 launches) that rely on S3 are also affected by this issue. Existing EC2 instances are unaffected by this issue. We will provide another update by 12:45 PM PST, or sooner if we have additional information to share.
4. Update · Mar 7, 19:53 UTC
   We are investigating increased error rates in the EU-CENTRAL-2 Region.