Development branches with MySQL Vectors not serving

Final update

All impacted branches are recovered. PlanetScale upgraded development Vitess branches with Vectors enabled from MySQL 8.0.x to 8.4.x. PlanetScale runs an internal fork containing support for PlanetScale's MySQL Vectors indexes. A bug in the 8.4.x version of our Vectors builds resulted in the upgrade immediately failing on startup. Because Vitess development branches only have one instance - the primary - impacted development branches did not have a primary during the impact window. The incident was resolved by reverting impacted branches to their previous MySQL version. PlanetScale will fix the bug in the MySQL Vectors 8.4.x build before re-attempting the upgrade at a later date.

1. Resolved · Jul 6, 21:45 UTC
   All impacted branches are recovered. PlanetScale upgraded development Vitess branches with Vectors enabled from MySQL 8.0.x to 8.4.x. PlanetScale runs an internal fork containing support for PlanetScale's MySQL Vectors indexes. A bug in the 8.4.x version of our Vectors builds resulted in the upgrade immediately failing on startup. Because Vitess development branches only have one instance - the primary - impacted development branches did not have a primary during the impact window. The incident was resolved by reverting impacted branches to their previous MySQL version. PlanetScale will fix the bug in the MySQL Vectors 8.4.x build before re-attempting the upgrade at a later date.
2. Monitoring · Jul 6, 20:38 UTC
   Branches are recovering and we are monitoring the situation.
3. Identified · Jul 6, 19:23 UTC
   We have identified the issue and are rolling impacted development branches from 8.4 back to 8.0.
4. Investigating · Jul 6, 19:18 UTC
   PlanetScale initiated a system wide upgrade of branches with Vectors enabled and running on MySQL 8.0 to MySQL 8.4. A number of development branches were disrupted and have not yet recovered. We are investigating and remediating.