Skip to content
Google Cloud · Cloud and hostingFeb 27, 2026, 12:37 UTC

Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

MinorConfig changeUpdated 18h ago
Feb 27, 12:37 UTCFeb 27, 14:35 UTC
Duration
1h 58m
Impact
Minor
Root cause
Config change
Google Cloud, 90 days
4 incidents
Affected
Agent AssistDialogflow CXVertex Gemini APIGoogle Cloud Supportglobalnorthamerica-northeast1southamerica-east1us-central1us-east1us-east4us-east5us-west1

What happened

Incident Report Summary On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes. This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability. Root Cause This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions. Remediation and Prevention Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation. Engineers identified the faulty configuration change a

Timeline

  1. available · Mar 9, 05:25 UTC
    Incident Report Summary On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes. This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability. Root Cause This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions. Remediation and Prevention Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation. Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. \ \ Google is committed to preventing a repeat of
  2. available · Mar 4, 23:23 UTC
    Preliminary Incident Report We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. Date/Time of the Issue (All time US/Pacific) Incident Start: 27 February 2026 04:37 Incident End: 27 February 2026 06:35 Duration: 1 hour, 58 minutes Summary On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models (including Gemini 2.0, 2.5, and 3.0 previews) experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, the Vertex Gemini API and Dialogflow CX in US regions and the global endpoint for a duration of 1 hour and 58 minutes. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Preliminary Root Cause This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. This configuration change enabled a code path that interacted poorly with specific requests, leading to service disruption for
  3. service_information · Feb 27, 16:12 UTC
    **Description** \ Between Friday, 2026-02-27, 04:36 and 06:45 PST, customers experienced increased error rates when accessing the Vertex Gemini API Global endpoint. The issue impacted API requests to multiple Gemini models. The incident also caused downstream impact to Dialogflow CX, Agent Assist, Google Cloud Support AI agent, and Customer Experience Agent Studio, which rely on Gemini APIs. Preliminary analysis indicates the issue was triggered by a recent configuration change. Service was fully restored after the configuration change was rolled back. We thank you for your patience while we worked on resolving the issue. **Symptom** \ Customers experienced increased error rates when sending API requests to impacted multiple Gemini models through the global endpoint.

More from Google Cloud

Full history

Also caused by configuration change

All
StartedIncidentDuration
Sep 2218:20 UTCPhone Number APIs and Console Were Returning Incorrect 404 ResponsesTwilio0m
Sep 1622:30 UTCHyperdrive Elevated Origin Connection Failure RatesCloudflare0m
Sep 1211:04 UTCSome customers experiencing blurry image previews and download issuesSlack8h 56m
Sep 1107:18 UTCINC20000213Snowflake6h 1m
Sep 407:34 UTCINC20000199Snowflake2h 36m
Sep 109:58 UTCINC20000190Snowflake3h 52m

From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.

Weekly: the week's major outages, postmortems and breaches, Saturday mornings.