Delays in event ingestion

Final update

This incident has been resolved.

1. Resolved · Jun 13, 13:52 UTC
   This incident has been resolved.
2. Monitoring · Jun 12, 22:15 UTC
   We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
3. Monitoring · Jun 11, 15:37 UTC
   We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
4. Monitoring · Jun 11, 05:14 UTC
   We have made partial progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
5. Monitoring · Jun 11, 04:12 UTC
   We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
6. Monitoring · Jun 11, 04:07 UTC
   We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
7. Monitoring · Jun 11, 04:07 UTC
   We are continuing to monitor for any further issues.
8. Monitoring · Jun 11, 01:31 UTC
   We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
9. Monitoring · Jun 10, 18:54 UTC
   We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.
10. Monitoring · Jun 9, 12:08 UTC
    We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.
11. Monitoring · Jun 9, 02:23 UTC
    We are continuing to monitor for any further issues.
12. Monitoring · Jun 9, 02:03 UTC
    We are monitoring the status of our services.
13. Monitoring · Jun 8, 21:24 UTC
    A fix has been implemented and we are monitoring the results.
14. Identified · Jun 8, 20:46 UTC
    We are currently experiencing delays in event ingestion impacting multiple product areas starting at 12:40pm PT. We have identified the issue and are working on a fix. No data has been lost.