Add a git merge driver for MODULE.bazel.lock#22428
Add a git merge driver for MODULE.bazel.lock#22428fmeum wants to merge 1 commit intobazelbuild:masterfrom
MODULE.bazel.lock#22428Conversation
578ca04 to
bbce622
Compare
| @@ -0,0 +1,54 @@ | |||
| # Merges an arbitrary number of MODULE.bazel.lock files. | |||
There was a problem hiding this comment.
@alexeagle for jq readability and "will devs like this"
|
@bazel-io fork 7.2.0 |
|
I'm very much not knowledgeable in git merge drivers or jq, so I'll leave the review to @alexeagle and just rubberstamp it when necessary. |
site/en/external/lockfile.md
Outdated
| 2. Run the following commands: | ||
|
|
||
| ```bash | ||
| jq_script=$(curl https://raw.githubusercontent.com/bazelbuild/bazel/release-7.2.0/scripts/bazel-lockfile-merge.jq) |
There was a problem hiding this comment.
I bet no one is going to maintain this line by updating for new releases, any ideas how to avoid that?
There was a problem hiding this comment.
The script only needs to change if we make a major change to the lockfile format, not on every bump to lockFileVersion, so this should be very rare and e.g. Bazel 7.3.0 would just reuse this link.
@Wyverald Is there any way to stamp "current Bazel version" into the docs? If not, could this be covered by // LINT.IfChange?
There was a problem hiding this comment.
@Wyverald Friendly ping.
I replaced this with a reference to master for now. If we can guarantee backwards compatibility indefinitely (for as long as MODULE.bazel.lock exists), that would be ideal as we can then register the script globally. Versioning it per project and Bazel version would be tedious anyway.
| ### Automatic Resolution {:#automatic-resolution} | ||
|
|
||
| Bazel provides a custom | ||
| [git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) |
There was a problem hiding this comment.
git merge drivers are clunky because of the setup steps.
I assume you considered having the same semantics as tools like npm which include the understanding of merge conflicts in their parser/resolver of the lockfile? Such that no external tooling would be needed?
There was a problem hiding this comment.
I agree that the one-time setup is clunky, but the usability after that is actually better than what npm offers: With npm, you have to resolve the conflict in package.json first and then run npm install. With this merge driver, you don't even get a conflict in MODULE.bazel.lock reported in the first place, regardless of what the conflict state of MODULE.bazel is.
We could theoretically build functionality into bazel that resolves the merge conflict in MODULE.bazel.lock even if MODULE.bazel contains an unresolved conflict, but that would either require a dedicated command or accepting that whatever Bazel command you run for this fail, but still updates the lockfile. That doesn't sound very discoverable.
Since only the initial setup is clunky, I hope that the contextual error message pointing to it when a merge conflict is observed will help make this usable overall.
What do you think about the tradeoff?
There was a problem hiding this comment.
Sorry I missed the reply.
whatever Bazel command you run for this fail, but still updates the lockfile
I think ideally a bazel build with conflicts in the lockfile would update the lockfile (maybe interactive prompt) and then succeed. Perhaps that can be built into one of the wrapper CLIs that already has interactivity.
Anyhow I really only meant to ask if you considered it, and not to block this being merged. Glad to see it's landed now!
|
@bazel-io fork 7.2.0 |
Adds a `jq` script to `scripts/` that merges any number of `MODULE.bazel.lock` files without using Bazel or reading the corresponding `MODULE.bazel` files.
The lockfile docs now have a section explaining the steps needed to set up this script as a custom merger driver for Git, which means that merge conflicts in `MODULE.bazel.lock` files will always be resolved automatically. Note that resolution may emit lockfiles with redundant information that will be dropped by subsequent Bazel invocations.
When Bazel encounters an error during lockfile parsing that could be caused by a merge conflict, it emits a different error message with a link to the docs. This required fixing the following kind of server crash when a conflict marker occurs inside a `recordedFileInputs` object:
```
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue$$Lambda/0x000000f8011da998@314cd9ee' (requested by nodes 'RegistryKey{url=https://bcr.bazel.build/}')
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:557)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:426)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.lang.IllegalArgumentException: the provided path should be absolute in the filesystem
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.createOutsideWorkspace(RepoRecordedInput.java:202)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.parse(RepoRecordedInput.java:222)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$File$1.parse(RepoRecordedInput.java:265)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:376)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:367)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:186)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:171)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:17)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:129)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:15)
at com.google.gson.Gson.fromJson(Gson.java:991)
at com.google.gson.Gson.fromJson(Gson.java:956)
at com.google.gson.Gson.fromJson(Gson.java:905)
at com.google.gson.Gson.fromJson(Gson.java:876)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.getLockfileValue(BazelLockFileFunction.java:93)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.compute(BazelLockFileFunction.java:73)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:468)
... 7 more
```
Alternatives considered:
* Letting Bazel resolve the conflict would require building knowledge about particular version control systems and their conflict style into Bazel. It would also either require the user to resolve conflicts in `MODULE.bazel` first or deviate from the current behavior that the lockfile is not updated when any Bzlmod error is encountered. The jq script can be used as is by every VCS with merge driver support and resolves the conflict in `MODULE.bazel.lock` independently of `MODULE.bazel`.
* Implementing the git merge driver as a `bazel mod` subcommand. This could be the source of intransparent slowdowns during regular git operations, which may even be triggered by other tools such as IDEs. The jq script is very fast.
* Implementing the merger as a Go binary in buildtools would replace the ubiquitous jq tool with a special purpose binary while also not solving the problem that per-user action is required once to register a custom merge driver.
Implements https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch
RELNOTES: Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup.
Closes bazelbuild#22428.
PiperOrigin-RevId: 640596606
Change-Id: I20659e3e53a7d8f2529f2ad5a3e7f258d7af026d
Adds a `jq` script to `scripts/` that merges any number of
`MODULE.bazel.lock` files without using Bazel or reading the
corresponding `MODULE.bazel` files.
The lockfile docs now have a section explaining the steps needed to set
up this script as a custom merger driver for Git, which means that merge
conflicts in `MODULE.bazel.lock` files will always be resolved
automatically. Note that resolution may emit lockfiles with redundant
information that will be dropped by subsequent Bazel invocations.
When Bazel encounters an error during lockfile parsing that could be
caused by a merge conflict, it emits a different error message with a
link to the docs. This required fixing the following kind of server
crash when a conflict marker occurs inside a `recordedFileInputs`
object:
```
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue$$Lambda/0x000000f8011da998@314cd9ee' (requested by nodes 'RegistryKey{url=https://bcr.bazel.build/}')
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:557)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:426)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.lang.IllegalArgumentException: the provided path should be absolute in the filesystem
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.createOutsideWorkspace(RepoRecordedInput.java:202)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.parse(RepoRecordedInput.java:222)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$File$1.parse(RepoRecordedInput.java:265)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:376)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:367)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:186)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:171)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:17)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:129)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:15)
at com.google.gson.Gson.fromJson(Gson.java:991)
at com.google.gson.Gson.fromJson(Gson.java:956)
at com.google.gson.Gson.fromJson(Gson.java:905)
at com.google.gson.Gson.fromJson(Gson.java:876)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.getLockfileValue(BazelLockFileFunction.java:93)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.compute(BazelLockFileFunction.java:73)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:468)
... 7 more
```
Alternatives considered:
* Letting Bazel resolve the conflict would require building knowledge
about particular version control systems and their conflict style into
Bazel. It would also either require the user to resolve conflicts in
`MODULE.bazel` first or deviate from the current behavior that the
lockfile is not updated when any Bzlmod error is encountered. The jq
script can be used as is by every VCS with merge driver support and
resolves the conflict in `MODULE.bazel.lock` independently of
`MODULE.bazel`.
* Implementing the git merge driver as a `bazel mod` subcommand. This
could be the source of intransparent slowdowns during regular git
operations, which may even be triggered by other tools such as IDEs. The
jq script is very fast.
* Implementing the merger as a Go binary in buildtools would replace the
ubiquitous jq tool with a special purpose binary while also not solving
the problem that per-user action is required once to register a custom
merge driver.
Implements
https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch
RELNOTES: Git merge conflicts in `MODULE.bazel.lock` files can be
resolved automatically. See
https://bazel.build/external/lockfile#automatic-resolution for the
required setup.
Closes #22428.
PiperOrigin-RevId: 640596606
Change-Id: I20659e3e53a7d8f2529f2ad5a3e7f258d7af026d
Commit
3187250
Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
|
Probably because the macOS VMs don't have Rosetta 2 installed. It it possible to upgrade jq to a version at least ships a macos arm64 binary? |
|
I will add one. |
|
I sent #18444 |
Adds a
jqscript toscripts/that merges any number ofMODULE.bazel.lockfiles without using Bazel or reading the correspondingMODULE.bazelfiles.The lockfile docs now have a section explaining the steps needed to set up this script as a custom merger driver for Git, which means that merge conflicts in
MODULE.bazel.lockfiles will always be resolved automatically. Note that resolution may emit lockfiles with redundant information that will be dropped by subsequent Bazel invocations.When Bazel encounters an error during lockfile parsing that could be caused by a merge conflict, it emits a different error message with a link to the docs. This required fixing the following kind of server crash when a conflict marker occurs inside a
recordedFileInputsobject:Alternatives considered:
MODULE.bazelfirst or deviate from the current behavior that the lockfile is not updated when any Bzlmod error is encountered. The jq script can be used as is by every VCS with merge driver support and resolves the conflict inMODULE.bazel.lockindependently ofMODULE.bazel.bazel modsubcommand. This could be the source of intransparent slowdowns during regular git operations, which may even be triggered by other tools such as IDEs. The jq script is very fast.Implements https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch
RELNOTES: Git merge conflicts in
MODULE.bazel.lockfiles can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup.