Spans, logs and metrics ingestion stopped
Jul 6, 13:30 UTCJul 6, 22:18 UTC
Duration
8h 48m
Impact
Informational
Root cause
Not disclosed
Sentry, 90 days
30 incidents
Affected
EU Ingestion - EU Span IngestionAPIDashboardUS Ingestion - Custom Metrics (beta)US Ingestion - US Span Ingestion
Final update
US ingestion is back to normal. EU ingestion is still delayed, but we are now recovering. We have created a new Statuspage incident for tracking EU, follow that for further updates: https://status.sentry.io/incidents/7tfjf5k3m0g3
Timeline
- Resolved · Jul 6, 22:18 UTC
US ingestion is back to normal. EU ingestion is still delayed, but we are now recovering. We have created a new Statuspage incident for tracking EU, follow that for further updates: https://status.sentry.io/incidents/7tfjf5k3m0g3
- Monitoring · Jul 6, 21:28 UTC
We're still burning the last of our backlog in US and expect to be back to normal the next hour. We're continuing to investigate the delayed ingestion in our EU region. EU API and dashboards are also affected by this incident.
- Monitoring · Jul 6, 20:29 UTC
We're continuing burning our backlog in US and expect to be have it fully consumed within the next hour. We're continuing to investigate the delayed ingestion in our EU region. EU API and dashboards are also affected by this incident.
- Monitoring · Jul 6, 19:26 UTC
We're continuing burning our backlog in US and expect to be have it fully consumed in about 1 hour. We're continuing to investigate the delayed ingestion in our EU region.
- Monitoring · Jul 6, 18:30 UTC
We've taken steps to mitigate the issue in the US and we're burning our backlog and expect to be have it fully consumed in about 4 hours. We're continuing to investigate the delayed ingestion in our EU region.
- Investigating · Jul 6, 17:14 UTC
We're continuing to investigate delayed ingestion of spans, logs, and metrics in both our US and EU regions.
- Investigating · Jul 6, 16:18 UTC
We're continuing to investigate delayed ingestion of spans, logs, and metrics in both our US and EU regions.
- Investigating · Jul 6, 15:22 UTC
We're continuing to investigate delayed ingestion of spans, logs, and metrics in both our US and EU regions.
- Investigating · Jul 6, 14:26 UTC
We're continuing to investigate delayed ingestion of spans, logs, and metrics in both our US and EU regions.
- Investigating · Jul 6, 13:30 UTC
We're continuing to investigate delayed ingestion of spans, logs, and metrics in both our US and EU regions.
- Monitoring · Jul 6, 05:48 UTC
We're still consuming the backlog and monitoring the situation.
- Monitoring · Jul 6, 02:07 UTC
Backlog is being consumed and we’re still monitoring recovery
- Monitoring · Jul 6, 00:44 UTC
The issue was fixed and we are now monitoring the fix
- Investigating · Jul 6, 00:35 UTC
We are investigating consumers for spans, logs and metrics are not ingested
More from Sentry
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 1810:17 UTC | Delays in spans ingestion | major | 41m |
| Sep 1808:36 UTC | Delayed ingestion of errors in EU | critical | 3h 20m |
| Sep 1016:13 UTC | Ingestion is delayed in US | minor | 57m |
| Sep 1015:25 UTC | Sentry.io elevated number of 500 errors | major | 1h 1m |
| Sep 114:16 UTC | Span ingestion is degraded in US | minor | 6h 14m |
| Sep 108:01 UTC | Sentry requests timing out / 504's | major | 3h 20m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.