-
Notifications
You must be signed in to change notification settings - Fork 4.4k
Bazel downloads unnecessary artifacts from remote cache #1922
Description
Consider this repository.
https://github.com/dfabulich/bazel-hazelcast-unnecessary-download
It includes a build file with three targets.
genrule(
name='big',
outs=['big.bin'],
cmd='dd if=/dev/urandom of=$@ count=100 bs=1048576',
)
genrule(
name='x',
srcs=['big'],
outs=['x.txt'],
cmd='echo hello > $@',
)
genrule(
name='y',
srcs=['x'],
outs=['y.txt'],
cmd='cp $(location x) $@',
)The big target generates a 100MB file. The x target depends on the big target and generates a short text file containing the word "hello". The y target depends on x and just copies it to the destination file.
This simulates the situation in our build, where we generate a large artifact as a build dependency for a much smaller artifact; downstream dependencies only need the smaller built file, not the large build dependency.
The tools/bazel.rc file in this repository contains settings that enable Hazelcast distributed artifact caching.
build --hazelcast_node=127.0.0.1:5701 --spawn_strategy=remote --genrule_strategy=remote
To reproduce the issue, clone the repository and run java -jar hazelcast-3.7.2.jar & to spawn Hazelcast in the background and run bazel build :y. Then run bazel clean && bazel build :y to fetch the built artifacts from the repository.
Actual: The critical path is 2s long; the 100MB big dependency is fetched and installed in bazel-genfiles/big.bin. (And it can take much longer than 2s if your dependency is even bigger than that, and if you're downloading the dependency over the internet.)
Expected: Since x only depends on big at build time, Bazel should skip downloading big and just download x.txt and provide it to y.txt.
If you comment out the srcs attribute of x, the build behaves as desired, and the critical path is 0.01s.
bazel version
$ bazel version
Build label: 0.3.2
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Oct 7 17:25:52 2016 (1475861152)
Build timestamp: 1475861152
Build timestamp as int: 1475861152