Elevated latencies for network change propagation

Final update

Between 7:08 AM and 10:48 AM PST, we experienced elevated latencies for network change propagation in the EU-WEST-1 Region. This was due to a delay in propagation of configuration updates on a sub-system which supports external and cross account connectivity. Customers experienced delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services were also impacted by this issue and were impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged at 6:29 AM, prior to customer impact beginning. At 7:20 AM, root cause was identified. By 9:20 AM we began observing early signs of recovery. Our mitigation efforts completed at 10:43 AM and full recovery was observed at 10:48 AM. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

1. Resolved · Jan 28, 19:22 UTC
   Between 7:08 AM and 10:48 AM PST, we experienced elevated latencies for network change propagation in the EU-WEST-1 Region. This was due to a delay in propagation of configuration updates on a sub-system which supports external and cross account connectivity. Customers experienced delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services were also impacted by this issue and were impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged at 6:29 AM, prior to customer impact beginning. At 7:20 AM, root cause was identified. By 9:20 AM we began observing early signs of recovery. Our mitigation efforts completed at 10:43 AM and full recovery was observed at 10:48 AM. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.
2. Update · Jan 28, 18:51 UTC
   We are seeing significant signs of recovery. We continue to work toward full resolution.
3. Update · Jan 28, 18:45 UTC
   We continue to work toward fully mitigating the issue that is resulting in elevated latencies for network change propagation delays in the EU-WEST-1 Region. We have a high degree of confidence that our current mitigation efforts will fully mitigate the issue. Our current estimation is that these mitigations will complete within the next 60 to 120 minutes. We recommend customers continue to retry failed requests. We will provide additional information as recovery continues to progress.
4. Update · Jan 28, 17:47 UTC
   We are seeing early signs of progressive recovery. We continue to work toward full recovery and will continue to provide updates as we work toward full mitigation. We recommend customers continue to retry failed requests. As recovery progresses, additional requests will succeed.
5. Update · Jan 28, 17:36 UTC
   We can confirm elevated latencies for network change propagation in the EU-WEST-1 Region. This causes delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services are also impacted by this issue and may be impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged and we are actively working on mitigating this issue. We plan to provide an update within the next 60 minutes.
6. Update · Jan 28, 17:14 UTC
   We are investigating elevated latencies for network change propagation in the EU-WEST-1 Region. This is resulting in impact to other services, such as timeouts when pulling images. We will provide you with further information shortly.