Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

migration: Add stitch utilities#36319

Merged
efritz merged 115 commits into
mainfrom
ef/stitch-migrations
Jul 26, 2022
Merged

migration: Add stitch utilities#36319
efritz merged 115 commits into
mainfrom
ef/stitch-migrations

Conversation

@efritz

@efritz efritz commented May 31, 2022

Copy link
Copy Markdown
Contributor

This PR adds utilities to combine the migration graphs over several releases of Sourcegraph so that we can perform upgrades over multiple versions where a migration file squash has occurred. Fixes #38045.

The new main method StitchDefinitions calculates a stitched migration graph for tests but is not yet utilized. Future PRs will bundle these results into artifacts that can be used to upgrade on-prem.

How to review - by file, in order:

  • stitch_test.go: Start with the test case to see the current coverage. There's currently only basic validation (is the graph valid), but we should also add testing to ensure the migration graph can run correctly against a live database in a subsequent PR. We validate the graph and the result of an upgrade from 3.29 -> 3.42.
  • git.go: Read these leaf functions that interact with the enclosing git clone to prep for the next two files.
  • reader.go: Read this file which reads "raw migrations" from the enclosing git clone.
  • rewriter.go: This is a wrapper around reading raw migrations (above) that does some specific rewrites to handle changes to migration definitions over time. Pay attention to details within this file, but forget them once moving on in the review. This file is how we support migrations defined back to 3.29.0 for all three schemas.
  • stitch.go: Read top-to-bottom with code flow to finish off how migrations are merged together.

Test plan

New unit tests introduced in this PR.

@efritz efritz self-assigned this May 31, 2022
@cla-bot cla-bot Bot added the cla-signed label May 31, 2022
@efritz efritz force-pushed the ef/stitch-migrations branch 3 times, most recently from 737fe75 to 681ab66 Compare June 7, 2022 23:02
@efritz efritz added the rfc-697 label Jun 30, 2022
@efritz efritz force-pushed the ef/stitch-migrations branch 2 times, most recently from cad8a6f to d5b6dfd Compare July 18, 2022 22:09
@efritz efritz force-pushed the ef/stitch-migrations branch 3 times, most recently from 285d578 to ae36092 Compare July 20, 2022 15:55
@efritz efritz force-pushed the ef/stitch-migrations branch 2 times, most recently from 35d9b80 to b5cb02c Compare July 20, 2022 18:29
@efritz efritz marked this pull request as ready for review July 20, 2022 18:29
@efritz efritz force-pushed the ef/stitch-migrations branch from 1abd6bc to 337dfcb Compare July 20, 2022 19:17
Piszmog and others added 9 commits July 26, 2022 11:11
This is the follow up/improved version of #39116

This commit adds support for syntax highlighting from lsif/scip data that is
returned for some languages. Updating the backend to return lsif/scip
data for every language is done in a separate PR (#39264), but this
PR does not depend on it (if treesitter highlighting is not configured
there simply won't be any highlighting).

Syntax highlighting is done by an extension that takes the JSON encoded
scip data and converts it to something CodeMirror can understand.
Decorations are only generated for the lines that are currently
rendered.

I originally converted line/column ranges to document-offset ranges and
used binary search to find the relevant ranges for the currently
rendered lines. However the overhead of doing the line/column -> offset
conversion was noticeable for large documents, but especially unnecessary
for those because only a small subset of lines would be visible.

Then I changed to an approach that would use the data sent from the
server as is (to avoid creating additional objects in memory and GC-ing
the JSON decoded data) and only add a simple line index. The conversion
from line/column to document offset is now delayed until the moment the
decorations are created. This only works under the assumption that the
server sends back the ranges order and without overlap.

Some of the changes I made (e.g. exporting the replaceValue function)
are not relevant anymore for the final version, but I'll leave them in
for completeness.

Other auxiliary changes in this commit:

Changed the base CodeMirror hook to allow for "manually" dispatching a
transaction to update the value. Without this we would trigger at
least two transactions when loading a file: One for updating the
document and one for updating the syntax highlighting.
Now we can do both in a single transaction.
Added a folder for CodeMirror blob extensions and moved the line numbers
extension there as well.
@sourcegraph-bot

sourcegraph-bot commented Jul 26, 2022

Copy link
Copy Markdown
Contributor

Codenotify: Notifying subscribers in OWNERS files for diff 1da629b...9dbe187.

Notify File(s)
@mrnugget dev/sg/sg_migration.go
@sourcegraph/dev-experience dev/sg/sg_migration.go

@efritz efritz enabled auto-merge (squash) July 26, 2022 16:44
@efritz efritz disabled auto-merge July 26, 2022 17:04
@efritz efritz enabled auto-merge (squash) July 26, 2022 18:15
@efritz efritz merged commit bdb58fd into main Jul 26, 2022
@efritz efritz deleted the ef/stitch-migrations branch July 26, 2022 18:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC 697: Support reconstruction of migration definitions over time