Skip to content
GitHub · Developer toolsDec 3, 2025, 22:30 UTC

Webhooks delivery degradation

InformationalCapacityUpdated 18h ago
Dec 3, 22:30 UTCDec 3, 22:30 UTC
Duration
0m
Impact
Informational
Root cause
Capacity
GitHub, 90 days
68 incidents
Affected
Not listed by the vendor.
Status page

Final update

On December 3, 2025, between 22:21 UTC and 23:44 UTC, the Webhooks service experienced a degradation that delayed writes of webhook delivery records to our database. During this period, many webhook deliveries were not visible in the webhook delivery UI or API for more than an hour after they were sent. As a result, customers were temporarily unable to request redeliveries for those delayed records. The underlying cause was throttling of database writes due to high replication lag. We mitigated the incident by temporarily disabling delivery history for a small number of very high‑volume webhook owners to reduce write pressure and stabilize the service. We are contacting the affected customers directly with more details. We are improving our webhook delivery storage architecture so it can scale with current and future webhook traffic, reducing the likelihood and impact of similar issues.

Timeline

  1. Resolved · Dec 3, 22:30 UTC
    On December 3, 2025, between 22:21 UTC and 23:44 UTC, the Webhooks service experienced a degradation that delayed writes of webhook delivery records to our database. During this period, many webhook deliveries were not visible in the webhook delivery UI or API for more than an hour after they were sent. As a result, customers were temporarily unable to request redeliveries for those delayed records. The underlying cause was throttling of database writes due to high replication lag. We mitigated the incident by temporarily disabling delivery history for a small number of very high‑volume webhook owners to reduce write pressure and stabilize the service. We are contacting the affected customers directly with more details. We are improving our webhook delivery storage architecture so it can scale with current and future webhook traffic, reducing the likelihood and impact of similar issues.

More from GitHub

Full history
StartedIncidentDuration
Sep 2310:11 UTCIncident across several servicesOngoing
Sep 2022:13 UTCIncident with Pull Requests1h 9m
Sep 1720:59 UTCElevated rate of errors for OpenAI models provided by Copilot50m
Sep 1607:20 UTCDegradation with Gemini 3.8 Flash10h 28m
Sep 1519:11 UTCDisruption with some GitHub services49m
Sep 1509:47 UTCDisruption with some GitHub services1h 30m

Also caused by capacity and load

All
StartedIncidentDuration
Sep 2213:19 UTCWe are investigating an issue with CH servers in Azure germanywestcentral regionClickHouse5h 6m
Sep 221:44 UTCElevated Linux worker queue timesExpo21m
Aug 2506:53 UTC[Critical] Issue with DownloadsBox44m
Aug 2015:40 UTCWe are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.Google Cloud3h 40m
Aug 1812:58 UTCINC20000163Snowflake1h 54m
Aug 1713:19 UTCNo capacity in ARNFly.io2h 22m

From vendors' own status pages and disclosures. Times as reported. Logos via logo.dev; trademarks belong to their owners.

Weekly: the week's major outages, postmortems and breaches, Saturday mornings.