Info for reviewers

In 5.1 we’ll be migrating the server image from Alpine to Wolfi, which means server's built-in Postgres database needs reindexing (see RFC 793).

The existing reindexing script is only used with the standalone database containers, so I’ve adapted it to work with server. I’ve tested the cases I can think of locally, but would appreciate a thorough review of the changes.

To get this running locally for testing, you can use the following images:

• Alpine: sourcegraph/server:5.0.5
• Wolfi: us.gcr.io/sourcegraph-dev/server:bazel-5201a0869411_2023-06-13 (built from this CI run)

Additional background

To be clear, only server (i.e. single-container deployments) will require a database reindex in 5.1; we won’t be migrating the standalone postgres-12-alpine, codeintel-db, codeinsights-db containers in this release.

The reindexing time varies a lot - it can be very quick, but for instances with lots of code intel it can be several hours. To avoid surprises the migration will not run unless the server container is started with the envar SOURCEGRAPH_5_1_DB_MIGRATION=true. Working with CE/TA to ensure all affected customers get a heads-up ahead of time.

When an admin first starts up a server instance after upgrading to 5.1, they'll see the following message:

**************** MIGRATION REQUIRED **************

Upgrading to Sourcegraph 5.1 or later from an earlier release requires a database reindex.

This process may take several hours, depending on the size of your database.

If you do not wish to perform the reindex process now, you should switch back to a release before Sourcegraph 5.1.

To perform the reindexing process now, please review the instructions at https://docs.sourcegraph.com/admin/migration/5_1 and restart the container with the environment variable `SOURCEGRAPH_5_1_DB_MIGRATION=true`
set.

**************** MIGRATION REQUIRED **************

Todo

• Ensure this correctly handles the case where an external database is used
• Ensure reindex.sh script can accommodate 3.31 and 5.1 migrations correctly
• Add 5.1 migration file when a new database is initialised on 5.1
• Clean up references to paths
• Test thoroughly
• Remove CI SKIP_DB_MIGRATION workaround

Test plan

I've tested the following cases locally using the following images:

• Alpine: sourcegraph/server:5.0.5
• Wolfi: us.gcr.io/sourcegraph-dev/server:bazel-5201a0869411_2023-06-13 (built from this CI run)

• Upgrading an existing Alpine server instance to this new Wolfi image
  • If envar is not set, upgrade is not run and migration info is shown
  • If envar is set, upgrade runs the first time the Wolfi image is started
• Downgrading is possible prior to migrating
  • If a user upgrades to 5.1 and sees the migration warning but does not start a migration, they can safely downgrade to their previous version with no additional work
• If a new database is initialised with the Wolfi image, no reindexing is required and envar is not required
• If a server instance is used with a standalone database, the reindexing is not performed (based on PGHOST)
• Updated reindex.sh script continues to work for standalone database containers
• Green main-dry-run: https://buildkite.com/sourcegraph/sourcegraph/builds/227511#0188b9c2-a544-4aa4-9328-7c74c90a1622

Local testing can be done by starting an Alpine-based instance locally with mounted data volumes and setting up Sourcegraph. Once done, stop the image and switch to a Wolfi-based image - this should perform a full reindexing process on startup.