Handle overriding symbolic links in pkg_install#1046
Merged
aiuto merged 1 commit intobazelbuild:mainfrom Mar 17, 2026
Merged
Conversation
The same way we handle overriding an existing file, we can override an existing symlink when installing
aiuto
approved these changes
Mar 17, 2026
|
|
||
| def _do_symlink(self, target, link_name, mode, user, group): | ||
| logging.debug("SYMLINK %s <- %s", link_name, target) | ||
| if os.path.lexists(link_name): |
Collaborator
There was a problem hiding this comment.
It's not obvious if we should always delete the object,
or only remove it was previously a link.
My hunch is that this behavior helps you find errors, even though it will fail in a variety of cases.
I can argue for either side.
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 19, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix a Windows breakage in compliance/package_licenses.bzl exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). ### Describe how you validated your changes Reproduced the failure locally on a Windows VM with rules_python 1.9.0, confirmed the fix resolves it.
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix a Windows breakage in compliance/package_licenses.bzl exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). ### Describe how you validated your changes Reproduced the failure locally on a Windows VM with rules_python 1.9.0, confirmed the fix resolves it.
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
What does this PR do? bump rules_pkg to Mar 17 main, picking up: - bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it - bazelbuild/rules_pkg#1035: affects reproducibiliy of builds - bazelbuild/rules_pkg#1044 - bazelbuild/rules_pkg#1046: a contribution of ours (@chouquette) - bazelbuild/rules_pkg#1047: we might need this
gh-worker-dd-mergequeue-cf854d bot
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump `rules_pkg` to current `main`. ### Motivation Pick up: - bazelbuild/rules_pkg#1021: a contribution of ours - bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it - bazelbuild/rules_pkg#1035: affects reproducibiliy of builds - bazelbuild/rules_pkg#1044 - bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette) - bazelbuild/rules_pkg#1047: we might need this feature Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix Windows breakages in compliance/package_licenses.bzl and deps/openssl.BUILD.bazel exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). The openssl FIPS build on Windows declares out_data_dirs = ["ssl", "lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir is set to an absolute path (C:/Program Files/…) outside the Bazel sandbox, so install_ssldirs writes there and leaves the sandbox copy empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows FIPS (the FIPS provider comes from @openssl_fips). The fix drops these from out_data_dirs on Windows FIPS via the existing fips_windows config_setting. ### Describe how you validated your changes Reproduced both failures locally on a Windows VM with rules_python 1.9.0, confirmed both fixes resolve them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix Windows breakages in compliance/package_licenses.bzl and deps/openssl.BUILD.bazel exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). The openssl FIPS build on Windows declares out_data_dirs = ["ssl", "lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir is set to an absolute path (C:/Program Files/…) outside the Bazel sandbox, so install_ssldirs writes there and leaves the sandbox copy empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows FIPS (the FIPS provider comes from @openssl_fips). The fix drops these from out_data_dirs on Windows FIPS via the existing fips_windows config_setting. ### Describe how you validated your changes Reproduced both failures locally on a Windows VM with rules_python 1.9.0, confirmed both fixes resolve them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix Windows breakages in compliance/package_licenses.bzl and deps/openssl.BUILD.bazel exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). The openssl FIPS build on Windows declares out_data_dirs = ["ssl", "lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir is set to an absolute path (C:/Program Files/…) outside the Bazel sandbox, so install_ssldirs writes there and leaves the sandbox copy empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows FIPS (the FIPS provider comes from @openssl_fips). The fix drops these from out_data_dirs on Windows FIPS via the existing fips_windows config_setting. ### Describe how you validated your changes Reproduced both failures locally on a Windows VM with rules_python 1.9.0, confirmed both fixes resolve them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes
added a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 20, 2026
### What does this PR do? Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite for bumping rules_foreign_cc to a commit that requires its transitive deps to be up-to-date. Fix Windows breakages in compliance/package_licenses.bzl and deps/openssl.BUILD.bazel exposed by the bump. ### Motivation rules_python 1.9.0 enables runfiles on Windows for py_binary (bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at the build-time runfiles directory, 1.9.0 makes bazel run copy the runfiles into a fresh temp directory. Bazel skips empty directories in that copy. package_licenses unconditionally included the offers_dir tree artifact in pkg_files. ship_source_offer is not declared by any dep, so offers_dir is always empty. On Linux and macOS bazel run uses symlinks, so an empty tree is harmless. On Windows the directory is absent from the temp copy, and pkg_install's copytree call fails with FileNotFoundError. The fix excludes offers_dir on Windows. The problem is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046 is unrelated). The openssl FIPS build on Windows declares out_data_dirs = ["ssl", "lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir is set to an absolute path (C:/Program Files/…) outside the Bazel sandbox, so install_ssldirs writes there and leaves the sandbox copy empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows FIPS (the FIPS provider comes from @openssl_fips). The fix drops these from out_data_dirs on Windows FIPS via the existing fips_windows config_setting. ### Describe how you validated your changes Reproduced both failures locally on a Windows VM with rules_python 1.9.0, confirmed both fixes resolve them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d bot
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 23, 2026
… misc files packaging (#47807) ### What does this PR do? This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries. * The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects. * If a version is provided, symlinks for that version will be created automatically * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time * A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install ### Motivation Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes. At the end of the day, we want to: * be able to automatically install all libraries and all their associated files * automatically patch the rpath when installing * automatically provide symlinks when installing * not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on. ### Describe how you validated your changes Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally ### Additional Notes This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this. RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward. All namings are debatable, no doc was provided until the API is settled. This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored) Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
StephenWakely
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 27, 2026
### What does this PR do? Bump `rules_pkg` to current `main`. ### Motivation Pick up: - bazelbuild/rules_pkg#1021: a contribution of ours - bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it - bazelbuild/rules_pkg#1035: affects reproducibiliy of builds - bazelbuild/rules_pkg#1044 - bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette) - bazelbuild/rules_pkg#1047: we might need this feature Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
StephenWakely
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 27, 2026
… misc files packaging (#47807) ### What does this PR do? This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries. * The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects. * If a version is provided, symlinks for that version will be created automatically * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time * A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install ### Motivation Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes. At the end of the day, we want to: * be able to automatically install all libraries and all their associated files * automatically patch the rpath when installing * automatically provide symlinks when installing * not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on. ### Describe how you validated your changes Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally ### Additional Notes This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this. RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward. All namings are debatable, no doc was provided until the API is settled. This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored) Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
StephenWakely
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 27, 2026
### What does this PR do? Bump `rules_pkg` to current `main`. ### Motivation Pick up: - bazelbuild/rules_pkg#1021: a contribution of ours - bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it - bazelbuild/rules_pkg#1035: affects reproducibiliy of builds - bazelbuild/rules_pkg#1044 - bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette) - bazelbuild/rules_pkg#1047: we might need this feature Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
StephenWakely
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Mar 27, 2026
… misc files packaging (#47807) ### What does this PR do? This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries. * The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects. * If a version is provided, symlinks for that version will be created automatically * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time * A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install ### Motivation Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes. At the end of the day, we want to: * be able to automatically install all libraries and all their associated files * automatically patch the rpath when installing * automatically provide symlinks when installing * not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on. ### Describe how you validated your changes Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally ### Additional Notes This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this. RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward. All namings are debatable, no doc was provided until the API is settled. This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored) Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The same way we handle overriding an existing file, we can override an existing symlink when installing
Fixes: #997