Web application unavailable and flag delivery evaluations have elevated failure rate

Final update

All service availability has been fully restored. Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it. To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.

1. Resolved · Jul 10, 18:19 UTC
   All service availability has been fully restored. Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it. To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.
2. Monitoring · Jul 10, 17:54 UTC
   Production service impact has been restored. However, customers might need to restart their applications to fully reconnect to LaunchDarkly's services. Our engineering team is working on guidance that can be provided to customers.
3. Identified · Jul 10, 17:40 UTC
   The web application availability has recovered. Flag delivery and event ingestion continue to be impacted and are actively recovering. Our engineering team continues to work on remediation.
4. Identified · Jul 10, 17:12 UTC
   We have identified the root cause and are implementing a fix.
5. Investigating · Jul 10, 17:10 UTC
   The LaunchDarkly web application (app.launchdarkly.com) was unavailable and is actively recovering at the moment. The Flag Delivery network was also impacted, causing an elevated failure rate with retrieving flag values. Customers using client-side or server-side SDKs should continue to see the last known flag values if a local cache exists, or fall back to in-code values. Customers using our Relay Proxy should continue to see last known flag values if a local cache exists. Customers using our Edge SDKs should continue to see last known flag values. Some event data loss may have occurred.