Google Cloud · Cloud and hostingJun 12, 2025, 17:49 UTC

Service Control crash loop returns 503s across Google Cloud

CriticalConfig changeUpdated 20h ago

Jun 12, 17:49 UTCJun 12, 20:49 UTC

Duration

Impact

Critical

Root cause

Config change

Google Cloud, 90 days

4 incidents

Affected

Service ControlCompute EngineBigQueryCloud StorageWorkspaceGlobal

Read the postmortem Status page

Lesson: New code paths belong behind feature flags, and globally replicated policy data needs staged propagation like any binary.

What happened

A policy change inserted into the regional Spanner tables that Service Control reads hit an unguarded code path with a null pointer, crashing Service Control binaries worldwide. External API requests returned 503 errors; us-central1 recovered last.

More from Google Cloud

Full history

Started	Incident	Impact	Duration
Sep 114:44 UTC	Multiple products in us-central1-b are experiencing network service degradation.	major	4h 8m
Aug 2015:40 UTC	We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.	critical	3h 40m
Jul 1523:57 UTC	Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.	major	12h 28m
Jul 1417:00 UTC	Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions.	major	10h 40m
Jun 507:00 UTC	Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.	major	21d 12h
Feb 2712:37 UTC	Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.	minor	1h 58m

Also caused by configuration change

All

Started	Vendor	Incident	Impact	Duration
Sep 2218:20 UTC	Twilio	Phone Number APIs and Console Were Returning Incorrect 404 ResponsesTwilio	none	0m
Sep 1622:30 UTC	Cloudflare	Hyperdrive Elevated Origin Connection Failure RatesCloudflare	none	0m
Sep 1211:04 UTC	Slack	Some customers experiencing blurry image previews and download issuesSlack	minor	8h 56m
Sep 1107:18 UTC	Snowflake	INC20000213Snowflake	critical	6h 1m
Sep 407:34 UTC	Snowflake	INC20000199Snowflake	critical	2h 36m
Sep 109:58 UTC	Snowflake	INC20000190Snowflake	major	3h 52m

From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.