Consider this repository.
https://github.com/dfabulich/bazel-hazelcast-unnecessary-download
It includes a build file with three targets.
genrule(
name='big',
outs=['big.bin'],
cmd='dd if=/dev/urandom of=$@ count=100 bs=1048576',
)
genrule(
name='x',
srcs=['big'],
outs=['x.txt'],
cmd='echo hello > $@',
)
genrule(
name='y',
srcs=['x'],
outs=['y.txt'],
cmd='cp $(location x) $@',
)
The big target generates a 100MB file. The x target depends on the big target and generates a short text file containing the word "hello". The y target depends on x and just copies it to the destination file.
This simulates the situation in our build, where we generate a large artifact as a build dependency for a much smaller artifact; downstream dependencies only need the smaller built file, not the large build dependency.
The tools/bazel.rc file in this repository contains settings that enable Hazelcast distributed artifact caching.
build --hazelcast_node=127.0.0.1:5701 --spawn_strategy=remote --genrule_strategy=remote
To reproduce the issue, clone the repository and run java -jar hazelcast-3.7.2.jar & to spawn Hazelcast in the background and run bazel build :y. Then run bazel clean && bazel build :y to fetch the built artifacts from the repository.
Actual: The critical path is 2s long; the 100MB big dependency is fetched and installed in bazel-genfiles/big.bin. (And it can take much longer than 2s if your dependency is even bigger than that, and if you're downloading the dependency over the internet.)
Expected: Since x only depends on big at build time, Bazel should skip downloading big and just download x.txt and provide it to y.txt.
If you comment out the srcs attribute of x, the build behaves as desired, and the critical path is 0.01s.
bazel version
$ bazel version
Build label: 0.3.2
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Oct 7 17:25:52 2016 (1475861152)
Build timestamp: 1475861152
Build timestamp as int: 1475861152
Consider this repository.
https://github.com/dfabulich/bazel-hazelcast-unnecessary-download
It includes a build file with three targets.
The
bigtarget generates a 100MB file. Thextarget depends on thebigtarget and generates a short text file containing the word "hello". Theytarget depends onxand just copies it to the destination file.This simulates the situation in our build, where we generate a large artifact as a build dependency for a much smaller artifact; downstream dependencies only need the smaller built file, not the large build dependency.
The
tools/bazel.rcfile in this repository contains settings that enable Hazelcast distributed artifact caching.To reproduce the issue, clone the repository and run
java -jar hazelcast-3.7.2.jar &to spawn Hazelcast in the background and runbazel build :y. Then runbazel clean && bazel build :yto fetch the built artifacts from the repository.Actual: The critical path is 2s long; the 100MB
bigdependency is fetched and installed inbazel-genfiles/big.bin. (And it can take much longer than 2s if your dependency is even bigger than that, and if you're downloading the dependency over the internet.)Expected: Since
xonly depends onbigat build time, Bazel should skip downloadingbigand just downloadx.txtand provide it toy.txt.If you comment out the
srcsattribute ofx, the build behaves as desired, and the critical path is 0.01s.bazel version