Postmortem library
- Multiple products in us-central1-b are experiencing network service degradation.
Google CloudSep 1, 2026 · 4h 8m
Teams should check service health dashboards when regional network service degradation affects cloud products.
- Incident with Copilot AI Model Providers
GitHubAug 27, 2026 · 2h 8m
Engineering teams should monitor provider health when using third party AI model services.
- Incident with Actions
Teams should monitor service status when experiencing issues with automated workflow systems.
- Disruption with some GitHub services
Teams should check platform status dashboards during general service disruptions.
- We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.
Teams should monitor regional health alerts when experiencing elevated latencies and errors across cloud products.
- Intermittent failures creating agent tasks
Teams should track agent task creation pipelines to handle intermittent failures promptly.
- Incident with GitHub.com
Engineering teams should subscribe to platform status updates to remain informed during service incidents.
- Incident with Actions
Teams should verify service health when experiencing disruptions with deployment actions.
- Incident with Pages - Deployment Lag
Teams should account for deployment lag by monitoring static site hosting service health.
- Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.
Teams should design multi region redundancy to withstand datacenter outages caused by cooling failures.
- Pull Requests and Issues unavailable for signed-out users
- Incident with Actions, we are investigating reports of degraded availability
- Disruption with Copilot chat and Copilot Coding Agent
- Partial degradation for code scanning default setup and for code quality
- Multiple services are affected, service degradation
- Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.
- Body parsing change for a React Server Components fix causes 25-minute outage
CloudflareDec 5, 2025 · 25m
Urgent security mitigations still need a staged rollout; speed of response is exactly when a global config push is most dangerous.
- Oversized Bot Management feature file breaks the core proxy
Treat internally generated config files like user input: validate size and shape before they propagate, and keep global kill switches for each feature.
- DynamoDB DNS failure takes down US-EAST-1
AWSOct 20, 2025 · 14h 32m
Automation that manages critical DNS needs its own guard against writing an empty record, and dependent services need to recover from a stale state on their own.
- Three infrastructure bugs intermittently degrade Claude responses
AnthropicAug 5, 2025 · 44d
Quality regressions hide in normal variance; continuous evaluations on production traffic catch what user reports cannot.
- 1.1.1.1 public DNS resolver unreachable for 62 minutes
Latent config errors can sit for weeks; progressive deployment and legacy-system cleanup matter as much as the change that finally triggers them.
- Workers KV storage failure cascades to Access, WARP and more
A shared internal primitive is a single point of failure for everything built on it; know which products have a hard dependency and give them a fallback.
- Service Control crash loop returns 503s across Google Cloud
New code paths belong behind feature flags, and globally replicated policy data needs staged propagation like any binary.
- R2 object storage disabled during a phishing report remediation
Abuse tooling needs the same guardrails as production changes: scope checks and a second pair of eyes before an action can disable a whole service.
- New telemetry service overwhelms Kubernetes control planes
OpenAIDec 11, 2024 · 4h 22m
Test changes at production cluster size, and keep break-glass access to the control plane that does not depend on the thing that is failing.
- Falcon Channel File 291 update crashes Windows hosts worldwide
CrowdStrikeJul 19, 2024 · 1h 18m
Content and configuration updates need the same staged rollout, validation and customer control as code releases.
- OS update breaks networking across regions
DatadogMar 8, 2023 · 26h 55m
Automatic updates are deployments: stagger them, and never let the same change land on every region in the same hour.
- Maintenance script deletes 883 customer sites
AtlassianApr 5, 2022 · 12d 16h
Deletion should be soft by default, and bulk restores need to be rehearsed at the scale of your largest possible mistake.
- Internal network congestion disrupts US-EAST-1
Retry storms turn a small change into congestion; clients need backoff, and monitoring must not share the network it monitors.
- Customer configuration triggers latent bug, 85% of network errors
FastlyJun 8, 2021 · 2h 48m
Customer configuration is untrusted input to a shared fleet; isolate its blast radius and test the bug classes it can reach.
- Overloaded AWS Transit Gateway takes Slack down on the first workday of 2021
SlackJan 4, 2021 · 2h 18m
Managed network components have scaling limits too; pre-warm for known traffic spikes and make sure provisioning survives the incident it is meant to fix.
- WAF regular expression exhausts CPU worldwide for 27 minutes
Rules and regexes are code; stage them, and use an engine with guaranteed linear time for untrusted input.
- Mistyped command removes S3 index servers in US-EAST-1
Tools should refuse to remove capacity below a safe minimum, and the status page cannot depend on the system it reports on.
- Primary database data accidentally deleted, 18-hour restore
GitLabJan 31, 2017 · 19h
A backup is only real once a restore has been tested; destructive commands on production hosts need an unmistakable prompt.
From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.