Problem:
After switching to rules_go v0.56.0, we started experiencing the following error:
[2025-09-09T21:15:50.534Z] (21:15:50) [ERROR: [0m/home/ubuntu/.cache/bazel/_bazel_ubuntu/fdc7e464f0a2b4a76bb07016cd725e2e/external/src/infra/service/joom-infra-image-service/BUILD.bazel:75:12: Export of joom-infra-image-service-indices to ../src/infra/service/joom-infra-image-service/joom-infra-image-service-mongo-indices.yaml with mongo-indices kind failed: (Exit 127): joom-infra-image-service failed: error executing ExportMongoIndices command (from target @@src//infra/service/joom-infra-image-service:joom-infra-image-service-indices)
[2025-09-09T21:15:50.534Z] (cd /home/ubuntu/.cache/bazel/_bazel_ubuntu/fdc7e464f0a2b4a76bb07016cd725e2e/execroot/_main && \
[2025-09-09T21:15:50.534Z] exec env - \
[2025-09-09T21:15:50.534Z] bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/src/infra/service/joom-infra-image-service/joom-infra-image-service --env localhost --param 'log.level=fatal' '--printAppInfo=false' --exportManifestKind mongo-indices --exportManifestFile bazel-out/k8-fastbuild-ST-27f7ed591fdb/bin/external/src/infra/service/joom-infra-image-service/joom-infra-image-service-mongo-indices.yaml --param 'countries.disableLoad=true' --param 'log.bufferStderr=false' --param 'payments.paypal.disabled=true' --param 'payments.paypal_usa.disabled=true' --param 'payments.qiwi.disabled=true' --param 'payments.yandex_h2h.disabled=true' '--forceStubRuntimeConfig=true')
[2025-09-09T21:15:50.535Z] # Configuration: 9db84b7f99f055d11de7f063ce43916f499608655caf4f5dc53df99eac1f5242
[2025-09-09T21:15:50.535Z] # Execution platform: //bazel/platforms:force_llvm
[2025-09-09T21:15:50.535Z] bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/src/infra/service/joom-infra-image-service/joom-infra-image-service: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
Some of our targets stopped resolving shared libraries.
We noticed the issue is related to changes introduced in PR #4390, which modified how rpaths are calculated.
Debug info and observation:
We patched both rules_go v0.51.1 and v0.56.0 to log the actual rpaths being calculated.
For v0.51.1:
rpaths =[
"$ORIGIN/../../../../../_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnu",
"$ORIGIN/_joom-infra-image-service.runfiles/_main/_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnu"
]
For v0.56.0:
rpaths = [
"$ORIGIN/../../../../_main/_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnu",
"$ORIGIN/_joom-infra-image-service.runfiles/_main/_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnu"
]
The difference is here:
- v0.51.1 →
$ORIGIN/../../../../../_solib_k8
- v0.56.0 →
$ORIGIN/../../../../_main/_solib_k8
For the problematic target, $ORIGIN is:
/home/ubuntu/.cache/bazel/_bazel_ubuntu/fdc7e464f0a2b4a76bb07016cd725e2e/execroot/_main/bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/src/infra/service/joom-infra-image-service
The actual directory with shared libraries is:
bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnu
-
With v0.51.1, $ORIGIN/../../../../../_solib_k8 resolves correctly to .../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8
-
With v0.56.0, $ORIGIN/../../../../_main/_solib_k8 resolves to .../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/_main/_solib_k8, which does not contain the shared libraries.
Shared libraries can be put:
- Inside
.../*.runfiles/_main/_solib_k8/... (when using bazel run/test)
- This works fine: both versions generate a valid rpath to this directory, we can see it in the rpaths above.
- Inside
.../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8/...
- This is where the problem occurs. We have a custom rule that executes actions via
ctx.actions.run, and those binaries expect to load libraries from this directory. In v0.56.0, the rpath is broken.
Conclusion:
It looks like PR #4390 may have missed supporting the second case above.
Could you please confirm whether this is an issue in rules_go, or if we might have a configuration problem on our side?
We’re happy to provide more details if needed.
Additional info:
Problem:
After switching to rules_go v0.56.0, we started experiencing the following error:
Some of our targets stopped resolving
shared libraries.We noticed the issue is related to changes introduced in PR #4390, which modified how rpaths are calculated.
Debug info and observation:
We patched both rules_go v0.51.1 and v0.56.0 to log the actual rpaths being calculated.
For v0.51.1:
For v0.56.0:
The difference is here:
$ORIGIN/../../../../../_solib_k8$ORIGIN/../../../../_main/_solib_k8For the problematic target, $ORIGIN is:
/home/ubuntu/.cache/bazel/_bazel_ubuntu/fdc7e464f0a2b4a76bb07016cd725e2e/execroot/_main/bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/src/infra/service/joom-infra-image-serviceThe actual directory with
shared librariesis:bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8/_U_A_Abzlmod~~sysroot_Uldd~sysroot_Uldd_Uamd64_S_Slibjemalloc_Clibjemalloc___Uusr_Slib_Sx86_U64-linux-gnuWith v0.51.1,
$ORIGIN/../../../../../_solib_k8resolves correctly to.../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8With v0.56.0,
$ORIGIN/../../../../_main/_solib_k8resolves to.../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/external/_main/_solib_k8, which does not contain theshared libraries.Shared libraries can be put:
.../*.runfiles/_main/_solib_k8/...(when using bazel run/test).../bazel-out/k8-opt-exec-ST-cd39ae64b419/bin/_solib_k8/...ctx.actions.run, and those binaries expect to load libraries from this directory. In v0.56.0, the rpath is broken.Conclusion:
It looks like PR #4390 may have missed supporting the second case above.
Could you please confirm whether this is an issue in rules_go, or if we might have a configuration problem on our side?
We’re happy to provide more details if needed.
Additional info: