Increased 5xx Errors

Final update

Between 12:45 AM and 4:18 AM PDT, we experienced increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Our engineers were automatically engaged and immediately began investigating the root cause. By 2:57 AM PDT, we identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections. At 3:52 AM PDT, we took multiple mitigation actions that led to to full recovery at 4:18 AM PDT. Now that the issue has been mitigated, customers who temporarily changed their origin type can safely revert these changes. Customers utilizing other origin types were not affected by this issue. The issue has been resolved and the service is operating normally.

1. Resolved · Jul 16, 12:21 UTC
   Between 12:45 AM and 4:18 AM PDT, we experienced increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Our engineers were automatically engaged and immediately began investigating the root cause. By 2:57 AM PDT, we identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections. At 3:52 AM PDT, we took multiple mitigation actions that led to to full recovery at 4:18 AM PDT. Now that the issue has been mitigated, customers who temporarily changed their origin type can safely revert these changes. Customers utilizing other origin types were not affected by this issue. The issue has been resolved and the service is operating normally.
2. Update · Jul 16, 11:57 UTC
   We continue to see significant signs of recovery as a result of our mitigation efforts, with full recovery expected within the next 45 minutes.
3. Update · Jul 16, 11:27 UTC
   We are seeing initial signs of recovery and continue to work toward full recovery.
4. Update · Jul 16, 11:16 UTC
   We continue working to resolve the increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Customers utilizing other origin types remain unaffected by this issue. We have further scoped the issue down to routing table capacity within the packet processing subsystem responsible for routing requests from CloudFront's edge locations to resources within customer VPCs. We have identified and are currently testing a mitigation strategy to resolve the issue. Once testing is complete, we will deploy the mitigation in a phased approach. Based on the results from these tests, we will provide a clearer estimated time for resolution in our next update. We continue to recommend that customers who are able to do so temporarily change their origin type to resolve the errors. We will provide another update by 5:15 AM PDT, or sooner if additional information becomes available.
5. Update · Jul 16, 10:18 UTC
   We continue working to resolve the increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Customers utilizing other origin types remain unaffected by this issue. Based on our investigation, we believe the root cause is related to a packet processing subsystem responsible for routing requests from CloudFront's edge locations to resources within customer VPCs. We continue to recommend that customers who are able to do so temporarily change their origin type to resolve the errors. We will provide another update by 4:15 AM PDT, or sooner if additional information becomes available.
6. Update · Jul 16, 09:21 UTC
   Starting at 12:45 AM PDT, we are experiencing increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. We have confirmed that customers utilizing other origin types are not impacted by this issue. Our engineers are engaged and are actively working to mitigate impact. As a workaround, customers who do not require VPC Origins can change their origin type to resolve the errors. We will provide another update by 3:15 AM PDT, or sooner if more information becomes available.
7. Update · Jul 16, 08:44 UTC
   We are investigating increased 5xx errors for Cloudfront customers utilizing VPC Origins connectivity.