Grafana Cloud Hosted Logs increased write latency in eu-west-3 region.
Oct 30, 09:30 UTCOct 30, 13:36 UTC
Duration
4h 6m
Impact
Minor
Root cause
Capacity
Grafana Labs, 90 days
75 incidents
Affected
Not listed by the vendor.
Final update
We consider the incident as resolved now. With regards to the cause - a slow physical partition of the backend database, which is used by the control plane of a critical component caused increased latency and occasional overloading with subsequent failing of the write path. Once writes switched to a different partition, the latency dropped and error rate went down.
Timeline
- Resolved · Oct 30, 13:36 UTC
We consider the incident as resolved now. With regards to the cause - a slow physical partition of the backend database, which is used by the control plane of a critical component caused increased latency and occasional overloading with subsequent failing of the write path. Once writes switched to a different partition, the latency dropped and error rate went down.
More from Grafana Labs
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2221:27 UTC | Kubernetes Observability Billing & Usage Incorrect | minor | 16h 39m |
| Sep 2214:57 UTC | IRM Access Issues for a Small Group of Users | major | 20h 10m |
| Sep 2119:16 UTC | Mimir write request errors | none | 0m |
| Sep 1712:34 UTC | Elevated Latency Managing Cloud Provider Integrations in GCP US Central | minor | 0m |
| Sep 1510:00 UTC | Increased Execution Time for Browser Checks in Synthetic Monitoring | minor | 1h 17m |
| Sep 1507:49 UTC | Intermittent Metric Write Errors in GCP US Central (prod-us-central-0) | minor | 0m |
Also caused by capacity and load
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 2213:19 UTC |  ClickHouse | We are investigating an issue with CH servers in Azure germanywestcentral regionClickHouse | minor | 5h 6m |
| Sep 1607:20 UTC |  GitHub | Degradation with Gemini 3.8 FlashGitHub | major | 10h 28m |
| Sep 1509:47 UTC | GitHub | Disruption with some GitHub servicesGitHub | minor | 1h 30m |
| Sep 422:02 UTC | GitHub | Degradation in repos contents APIGitHub | minor | 21m |
| Sep 221:44 UTC |  Expo | Elevated Linux worker queue timesExpo | minor | 21m |
| Sep 115:00 UTC | GitHub | Delays in commit processingGitHub | minor | 1h 1m |
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.