Avoid action conflicts due to different configuration hashes #14236
Description
Description of the problem / feature request:
Bazel errors out if the same action (with identical action hash) is created with different configuration hash. Configuration hash should be irrelevant when considering if two actions are identical.
The graph in the minimal example below shows the issue where B and C sets different values of a build setting that controls the output of E:
A -> B -> D -> E
A -> C -> D -> E
Case 1: Using bazel with current ST-folders on above graph yields duplicated actions for target D:
$ bazel build //:A1
bazel-out/k8-fastbuild-ST-49bba2382c95/bin/libE1_bs1_B.a
bazel-out/k8-fastbuild-ST-49bba2382c95/bin/libD1.a
bazel-out/k8-fastbuild-ST-1f07a369810f/bin/libE1_bs1_C.a
bazel-out/k8-fastbuild-ST-1f07a369810f/bin/libD1.a
Case 2: To avoid duplicated actions in D, we can control the "output directory name" through transitions and thus remove the ST-hash. E avoids action conflicts by consider the build setting value in it's output path. This produces the the error in the topic.
$ bazel build //:A2
ERROR: file '_objs/D2/D2.pic.o' is generated by these conflicting actions:
Label: //:D2
RuleClass: library_input_outdir rule
Configuration: 169a94e6fa80a9d7d69db38a12e45a0f126e39b4e60d76c09ff240fd4812eddb, 9ae0f89835b3c1d0ad9e25dcc04eeb78120ba952c193e68b4a176c42a2734fd0
Mnemonic: CppCompile
Action key: 83337fb9f28e5353fb2871bf039e7dcdb7c41898cabc4b205d23cc7961d232b2
Progress message: Compiling D2.c
PrimaryInput: File:[[<execution_root>]bazel-out/any/bin]D2.c
PrimaryOutput: File:[[<execution_root>]bazel-out/any/bin]_objs/D2/D2.pic.o
Owner information: ConfiguredTargetKey{label=//:D2, config=BuildConfigurationKey[169a94e6fa80a9d7d69db38a12e45a0f126e39b4e60d76c09ff240fd4812eddb]}, ConfiguredTargetKey{label=//:D2, config=BuildConfigurationKey[9ae0f89835b3c1d0ad9e25dcc04eeb78120ba952c193e68b4a176c42a2734fd0]}
MandatoryInputs: are equal
Outputs: are equal
ERROR: com.google.devtools.build.lib.skyframe.ArtifactConflictFinder$ConflictException: com.google.devtools.build.lib.actions.MutableActionGraph$ActionConflictException: for _objs/D2/D2.pic.o, previous action: action 'Compiling D2.c', attempted action: action 'Compiling D2.c'
INFO: Elapsed time: 0.048s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
Note that his works for bazel 4.x but this functionality was dropped in af0e20f. Later versions need a revert of that commit to work.
Case3: Today we need, as a workaround, have the same value of the conflicting build setting in all paths to D. In this example we reset the value to default value at the input of D. This has the unwanted consequence that E can no longer be a dependency of D.
$ bazel build //:A3
bazel-out/any/bin/libE3_bs1_default.a
bazel-out/any/bin/libD3.a
Both versions of E are gone and both B and C links to an unintended default version.
To workaround 3, we have to rearrange the dependency graph in a non-optimal way, possibly with more targets than desired.
Feature requests: what underlying problem are you trying to solve with this feature?
To allow functionality as explained in the example without duplicated actions or action conflicts.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Start with an empty directory and run the following script inside it. For usage, see description above.
config_hash_action_conflict_example_setup.sh.txt
touch WORKSPACE
cat <<EOF > BUILD.bazel
load(":rules.bzl", "build_setting", "library_no_transition", "library_input_outdir", "library_input_outdir_bs1")
load(":setup_graph.bzl", "setup_graph")
build_setting(
name = "bs1",
build_setting_default = "bs1_default",
)
[
setup_graph(case=case, rule=rule, rule_args=rule_args)
for case, rule, rule_args in [
("1", library_no_transition, {}),
("2", library_input_outdir, {
"input_settings": {
"//command_line_option:output directory name": "any",
}},
),
("3", library_input_outdir_bs1, {
"input_settings": {
"//command_line_option:output directory name": "any",
"//:bs1": "bs1_default",
}},
),
]
]
EOF
cat <<EOF > setup_graph.bzl
load(":rules.bzl", "no_transition", "input_bs1", "library_input_bs1")
def setup_graph(*, case, rule, rule_args):
no_transition(
name = "A" + case,
deps = [":B" + case, ":C" + case],
)
input_bs1(
name = "B" + case,
deps = [":D" + case],
input_settings = {"//:bs1": "bs1_B"},
)
input_bs1(
name = "C" + case,
deps = [":D" + case],
input_settings = {"//:bs1": "bs1_C"},
)
rule(
name = "D" + case,
deps = [":E" + case],
**rule_args
)
library_input_bs1(
name = "E" + case,
path_resolution = ["//:bs1"],
)
EOF
cat <<EOF > rules.bzl
load("@bazel_tools//tools/cpp:toolchain_utils.bzl", "find_cpp_toolchain")
BuildSettingInfo = provider(fields = ["value"])
def _build_setting_impl(ctx):
return BuildSettingInfo(value = ctx.build_setting_value)
def _dummy_rule_impl(ctx, files = []):
transitive = [dep[DefaultInfo].files for dep in ctx.attr.deps]
return [DefaultInfo(files = depset(direct = files, transitive = transitive))]
def _library_rule_impl(ctx):
path_resolution = "".join(["_%s" % dep[BuildSettingInfo].value for dep in ctx.attr.path_resolution])
name = ctx.label.name + path_resolution
src = ctx.actions.declare_file(name + ".c")
ctx.actions.write(src, "int %s() {return 1;}\n" % name)
cc_toolchain = find_cpp_toolchain(ctx)
feature_configuration = cc_common.configure_features(ctx = ctx, cc_toolchain = cc_toolchain)
cc_args = {"actions": ctx.actions, "feature_configuration": feature_configuration, "cc_toolchain": cc_toolchain}
compilation_context, compilation_outputs = cc_common.compile(
name = name, srcs = [src], **cc_args)
linking_context, linking_outputs = cc_common.create_linking_context_from_compilation_outputs(
name = name, compilation_outputs = compilation_outputs, **cc_args)
library_to_link = linking_outputs.library_to_link
files = []
files += [library_to_link.static_library] if library_to_link.static_library else []
files += [library_to_link.pic_static_library] if library_to_link.pic_static_library else []
return _dummy_rule_impl(ctx, files = files) + [CcInfo(
compilation_context = compilation_context, linking_context = linking_context)]
def _input_transition_impl(settings, attr):
settings = {setting: settings[setting] for setting in attr._input_settings}
settings.update(attr.input_settings)
return settings
def _create_custom_rule(implementation, input = [], cpp = False):
attrs = {"deps": attr.label_list(), "path_resolution": attr.label_list()}
args = {"attrs": attrs, "implementation": implementation}
if cpp:
args["fragments"] = ["cpp"]
args["toolchains"] = ["@bazel_tools//tools/cpp:toolchain_type"]
attrs["_cc_toolchain"] = attr.label(default = "@bazel_tools//tools/cpp:current_cc_toolchain")
if input:
attrs.update({
"input_settings": attr.string_dict(),
"_input_settings": attr.string_list(default = input),
"_whitelist_function_transition": attr.label(
default = "@bazel_tools//tools/whitelists/function_transition_whitelist"),
})
args["cfg"] = transition(inputs = input, outputs = input, implementation = _input_transition_impl)
return rule(**args)
build_setting = rule(
implementation = _build_setting_impl,
build_setting = config.string(flag = False),
)
outdir = "//command_line_option:output directory name"
no_transition = _create_custom_rule(_dummy_rule_impl)
input_bs1 = _create_custom_rule(_dummy_rule_impl, ["//:bs1"])
library_no_transition = _create_custom_rule(_library_rule_impl, cpp = True)
library_input_bs1 = _create_custom_rule(_library_rule_impl, ["//:bs1"], cpp = True)
library_input_outdir = _create_custom_rule(_library_rule_impl, [outdir], cpp = True)
library_input_outdir_bs1 = _create_custom_rule(_library_rule_impl, [outdir, "//:bs1"], cpp = True)
EOF
What operating system are you running Bazel on?
Ubuntu 20.04
What's the output of bazel info release?
development version
If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.
Used current HEAD from master branch on github
$ git checkout e8a066e
$ git revert af0e20f
$ # fix conflicts
$ git revert --continue
What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?
$ git remote get-url bazelbuild; git rev-parse bazelbuild/master ; git rev-parse HEAD ; git rev-parse HEAD~1
https://github.com/bazelbuild/bazel.git
e8a066e
4d2f7a74ead44aecf80f1b0b271f2b9fa2816d01
e8a066e
Have you found anything relevant by searching the web?
Any other information, logs, or outputs that you want to share?
Similar to #14023, solution to this issue should make #13587 obsolete.