Description of the problem:

When an executable has dynamically-linked dependencies that have transitions, shared libraries aren't always placed where the binary expects them in runtime. This means, for example, that such tests cannot be run in remote execution.

The rpath where shared libraries with transitions are looked up is $ORIGIN/_solib_k8/../../../k8-fastbuild-ST-<...>/bin/_solib_k8, but this path is not always available:

1. Locally, bazel-out/k8-fastbuild/bin/_solib_k8/ may not always exist (e.g. if all shared libraries have transitions). ld doesn't seem to follow ../ from nonexistent paths.
2. In remote execution (or when deploying a target by copying its runfiles directory), the configuration directory k8-fastbuild-ST-<...> isn't available at all. The right files exist under _solib_k8, but because of the rpath, ld doesn't look for them there.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Run bazel test :b on this: https://gist.github.com/quval/5077b2edc902e9dc0bcad2311a245ffa
(Or change cc_test to cc_binary with linkstatic = 0 and use bazel run).

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

4.1.0 (also happens at master)

Have you found anything relevant by searching the web?

Issue #9357 seems to be related (though Bazel doesn't crash here).

Any other information, logs, or outputs that you want to share?

After a90c408 the same library may not be linked more than once, so a solution may be to always add the default solib directory to the rpath. (When running locally, the wrong version may then be loaded for libraries that have been built for more than one configuration – this is already a potential issue; the default solib may have to go last if added as a fallback – or the configuration could be encoded in the mangled name of the library to prevent this altogether.) I'll be happy to send a PR if any of this makes sense.