Notification delivery
Jun 11, 09:50 UTCJun 11, 14:48 UTC
Duration
4h 58m
Impact
Informational
Root cause
Dependency
Sentry, 90 days
30 incidents
Affected
Notification Delivery - SlackNotification Delivery - EmailNotification Delivery - PagerDutyNotification Delivery - Microsoft Teams
Final update
This issue has been resolved.
Timeline
- Resolved · Jun 11, 14:48 UTC
This issue has been resolved.
- Monitoring · Jun 11, 14:18 UTC
The issue involving some alerts not firing has been mitigated. We will continue to monitor the situation to ensure it does not reoccur.
- Monitoring · Jun 11, 13:23 UTC
We have identified an issue with one of our cloud providers that might have caused some alert notifications to not fire starting at around 4:00 AM GMT+2. Since around 1:30 PM GMT+2, alert notifications are now fully operational in the US region, while delivery is still partially degraded in the EU region.
- Monitoring · Jun 11, 13:13 UTC
Notifications delivery is now close to fully functional in US and DE. We have identified the root cause as caused by one of our cloud providers, and are closely monitoring the situation.
- Investigating · Jun 11, 11:51 UTC
We are continuing to investigate the degraded notifications.
- Investigating · Jun 11, 10:53 UTC
We are continuing to investigate the degraded notifications.
- Investigating · Jun 11, 09:50 UTC
We're currently investigating degraded notification delivery.
More from Sentry
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 1810:17 UTC | Delays in spans ingestion | major | 41m |
| Sep 1808:36 UTC | Delayed ingestion of errors in EU | critical | 3h 20m |
| Sep 1016:13 UTC | Ingestion is delayed in US | minor | 57m |
| Sep 1015:25 UTC | Sentry.io elevated number of 500 errors | major | 1h 1m |
| Sep 114:16 UTC | Span ingestion is degraded in US | minor | 6h 14m |
| Sep 108:01 UTC | Sentry requests timing out / 504's | major | 3h 20m |
Also caused by third-party dependency
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1609:48 UTC |  DigitalOcean | Support Portal Service DisruptionDigitalOcean | minor | 11h 21m |
| Sep 1608:31 UTC |  Cloudflare | Support Helpdesk Availability IssuesCloudflare | minor | 6h 27m |
| Sep 417:03 UTC |  Grafana Labs | Degradation of Hosted Grafana in US Central RegionGrafana Labs | major | 3h 39m |
| Sep 416:05 UTC |  Expo | Elevated Linux worker queue timesExpo | major | 15h 4m |
| Sep 115:25 UTC | Grafana Labs | Investigating issues in US Central (prod-us-central-0, prod-us-central-5)Grafana Labs | minor | 4h 54m |
| Aug 3106:26 UTC |  Fly.io | Packet loss in ORDFly.io | minor | 1h 44m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.