Errors loading the Asana web app for some users in US

Final update

**Incident**: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal. ‌ **Impact**: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost. ‌ **Moving forward**: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access. ‌ Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

1. Postmortem · Jul 8, 21:28 UTC
   **Incident**: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal. ‌ **Impact**: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost. ‌ **Moving forward**: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access. ‌ Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
2. Resolved · Jul 7, 19:24 UTC
   This incident has been resolved.
3. Monitoring · Jul 7, 19:21 UTC
   We are continuing to monitor for any further issues.
4. Monitoring · Jul 7, 19:12 UTC
   We have implemented a mitigation and are monitoring to ensure recovery.
5. Identified · Jul 7, 18:49 UTC
   We are seeing errors for part (about 10%) of users in US for the Asana webapp. We've identified the root cause and are working on identifying a resolution.