Connectivity Issues
Final update
Between 3:55 AM and 4:15 AM PDT, we experienced connectivity issues that impacted connectivity to the US-WEST-2 Region. This impacted multiple AWS services in the Region. Some customers may have also experienced issues accessing the AWS Management Console, with connection timeouts and unresponsive pages. Connectivity within the Region was not affected. Our engineers were automatically engaged at 4:01 AM PDT, and immediately began investigating this issue. We identified the root cause as an issue with networking devices responsible for network routing from the Region to the Seattle Metro, and began working in parallel on multiple paths to mitigate the impact. We took mitigation measures that led to initial recovery at 4:15 AM PDT. As the network continued to stabilize following our mitigation actions, a brief reconvergence event occurred between 4:47 AM and 4:59 AM PDT. During this reconvergence period, some customers may have experienced intermittent connectivity issues to the Region as network routes were re-established. By 4:59 AM PDT, all routes had been fully restored and service metrics returned to pre-event levels. Customers using AWS Direct Connect through EqSe2, Westin Bui
Timeline
- Resolved · Jul 24, 13:01 UTC
Between 3:55 AM and 4:15 AM PDT, we experienced connectivity issues that impacted connectivity to the US-WEST-2 Region. This impacted multiple AWS services in the Region. Some customers may have also experienced issues accessing the AWS Management Console, with connection timeouts and unresponsive pages. Connectivity within the Region was not affected. Our engineers were automatically engaged at 4:01 AM PDT, and immediately began investigating this issue. We identified the root cause as an issue with networking devices responsible for network routing from the Region to the Seattle Metro, and began working in parallel on multiple paths to mitigate the impact. We took mitigation measures that led to initial recovery at 4:15 AM PDT. As the network continued to stabilize following our mitigation actions, a brief reconvergence event occurred between 4:47 AM and 4:59 AM PDT. During this reconvergence period, some customers may have experienced intermittent connectivity issues to the Region as network routes were re-established. By 4:59 AM PDT, all routes had been fully restored and service metrics returned to pre-event levels. Customers using AWS Direct Connect through EqSe2, Westin Building Exchange, Seattle experienced an extended impact window from 3:55 AM to 5:12 AM PDT. These customers would have experienced connectivity issues until network routes for this specific path were fully restored at 5:12 AM PDT. Customers connected redundantly through other AWS Direct Connect locat
- Update · Jul 24, 12:30 UTC
We continue to see significant signs of recovery as a result of our mitigation efforts for the connectivity issues impacting multiple AWS services in the US-WEST-2 Region. We have identified the root cause as an issue with a networking device responsible for network routing from the Region to the Seattle Metro. Engineers have finished all mitigation work. As routes continue to be restored, customers should see a continued reduction in error rates and timeouts when connecting to affected services. We are monitoring recovery progress closely and will continue working until all routes have been fully restored and service metrics return to pre-event levels. We will provide another update in the next 30-45 minutes.
- Update · Jul 24, 11:56 UTC
We are seeing initial signs of recovery and continue to work toward full recovery.
- Update · Jul 24, 11:40 UTC
We are investigating connecivity issues impacting multiple AWS services in the US-WEST-2 Region.
More from AWS
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2123:26 UTC | Increased Error Rates | minor | 58m |
| Sep 321:49 UTC | Increased API Error Rates | minor | 2h 18m |
| Aug 2102:02 UTC | Increased Error Rates | minor | 38m |
| Aug 1915:15 UTC | Increased Error Rates | minor | 3h 32m |
| Aug 1503:42 UTC | Increased Packet loss | minor | 3d |
| Jul 3117:33 UTC | Elevated Packet Loss | minor | 1h 21m |
Also caused by network
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1901:36 UTC | Elevated errors in Ashburn, VA (IAD) | none | 0m | |
| Sep 1411:59 UTC | Increased wait times for macOS jobs | minor | 1h | |
| Sep 410:57 UTC | Increased errors on High Performance Edge Network - FRA Region | minor | 0m | |
| Sep 214:18 UTC | Upstream network issues | minor | 1h 39m | |
| Sep 116:11 UTC | Service degradation in GCP us-central-1 | none | 19m | |
| Sep 114:44 UTC | Multiple products in us-central1-b are experiencing network service degradation. | major | 4h 8m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.