Increased Error Rates and Latencies

Final update

Between 1:42 PM and 2:25 PM PDT we experienced increased error rates and latencies for EC2 APIs in the EU-NORTH-1 Region. This issue also affected new instance launches. Other AWS Services that launch new instances or call EC2 APIs as part of their workflows were also affected by this issue. Existing EC2 instances were unaffected by this issue. During this time, customers would have received an Internal Server Error in the Management Console and APIs. Engineers were automatically engaged and began investigating the root cause. We identified the root cause as a planned configuration change. This change was reverted and we began observing recovery at 2:19 PM. By 2:25 PM, the issue was fully mitigated. We do not expect this issue to reoccur. Since the issue was mitigated at 2:25 PM, we have been processing a backlog for ELB workflows and expect this backlog to complete within the next 30 minutes. We recommend customers retry requests that failed during this time. The issue has been resolved and all services are operating normally.

1. Resolved · Jun 30, 21:53 UTC
   Between 1:42 PM and 2:25 PM PDT we experienced increased error rates and latencies for EC2 APIs in the EU-NORTH-1 Region. This issue also affected new instance launches. Other AWS Services that launch new instances or call EC2 APIs as part of their workflows were also affected by this issue. Existing EC2 instances were unaffected by this issue. During this time, customers would have received an Internal Server Error in the Management Console and APIs. Engineers were automatically engaged and began investigating the root cause. We identified the root cause as a planned configuration change. This change was reverted and we began observing recovery at 2:19 PM. By 2:25 PM, the issue was fully mitigated. We do not expect this issue to reoccur. Since the issue was mitigated at 2:25 PM, we have been processing a backlog for ELB workflows and expect this backlog to complete within the next 30 minutes. We recommend customers retry requests that failed during this time. The issue has been resolved and all services are operating normally.
2. Update · Jun 30, 21:29 UTC
   We are seeing early signs of recovery and continue to work toward full recovery.
3. Update · Jun 30, 21:21 UTC
   We can confirm increased error rates for the EC2 APIs, as well as errors launching new EC2 instances in the EU-NORTH-1 Region. Other AWS Services that launch new instances or call the EC2 APIs as part of their workflows may also be affected by this issue. During this time, customers may receive an Internal Server Error in the Management Console and APIs. Engineers were automatically engaged and began investigating the issue. We are actively working on identifying the root cause. Existing instances are unaffected by this issue. We will provide an update by 3:15 PM, or sooner if we have additional information to share.
4. Update · Jun 30, 21:02 UTC
   We are investigating increased launch errors and API errors in the EU-NORTH-1 Region. Existing instances are not affected by this issue.