Skip to content

Handle overriding symbolic links in pkg_install#1046

Merged
aiuto merged 1 commit intobazelbuild:mainfrom
chouquette:chouquette/allow_symlink_override
Mar 17, 2026
Merged

Handle overriding symbolic links in pkg_install#1046
aiuto merged 1 commit intobazelbuild:mainfrom
chouquette:chouquette/allow_symlink_override

Conversation

@chouquette
Copy link
Copy Markdown
Contributor

@chouquette chouquette commented Mar 16, 2026

The same way we handle overriding an existing file, we can override an existing symlink when installing

Fixes: #997

The same way we handle overriding an existing file, we can override an
existing symlink when installing
@aiuto aiuto changed the title fix(pkg_install): handle overriding symbolic links Handle overriding symbolic links in pkg_install Mar 17, 2026

def _do_symlink(self, target, link_name, mode, user, group):
logging.debug("SYMLINK %s <- %s", link_name, target)
if os.path.lexists(link_name):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not obvious if we should always delete the object,
or only remove it was previously a link.
My hunch is that this behavior helps you find errors, even though it will fail in a variety of cases.
I can argue for either side.

@aiuto aiuto merged commit 14c78b8 into bazelbuild:main Mar 17, 2026
6 checks passed
@chouquette chouquette deleted the chouquette/allow_symlink_override branch March 18, 2026 07:23
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 19, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix a Windows breakage in compliance/package_licenses.bzl exposed by
the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

### Describe how you validated your changes
Reproduced the failure locally on a Windows VM with rules_python 1.9.0,
confirmed the fix resolves it.
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix a Windows breakage in compliance/package_licenses.bzl exposed by
the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

### Describe how you validated your changes
Reproduced the failure locally on a Windows VM with rules_python 1.9.0,
confirmed the fix resolves it.
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
What does this PR do?
bump rules_pkg to Mar 17 main, picking up:
- bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it
- bazelbuild/rules_pkg#1035: affects reproducibiliy of builds
- bazelbuild/rules_pkg#1044
- bazelbuild/rules_pkg#1046: a contribution of ours (@chouquette)
- bazelbuild/rules_pkg#1047: we might need this
gh-worker-dd-mergequeue-cf854d bot pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump `rules_pkg` to current `main`.

### Motivation
Pick up:
- bazelbuild/rules_pkg#1021: a contribution of ours
- bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it
- bazelbuild/rules_pkg#1035: affects reproducibiliy of builds
- bazelbuild/rules_pkg#1044
- bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette)
- bazelbuild/rules_pkg#1047: we might need this feature

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix Windows breakages in compliance/package_licenses.bzl and
deps/openssl.BUILD.bazel exposed by the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

The openssl FIPS build on Windows declares out_data_dirs = ["ssl",
"lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir
is set to an absolute path (C:/Program Files/…) outside the Bazel
sandbox, so install_ssldirs writes there and leaves the sandbox copy
empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows
FIPS (the FIPS provider comes from @openssl_fips). The fix drops these
from out_data_dirs on Windows FIPS via the existing fips_windows
config_setting.

### Describe how you validated your changes
Reproduced both failures locally on a Windows VM with rules_python 1.9.0,
confirmed both fixes resolve them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix Windows breakages in compliance/package_licenses.bzl and
deps/openssl.BUILD.bazel exposed by the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

The openssl FIPS build on Windows declares out_data_dirs = ["ssl",
"lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir
is set to an absolute path (C:/Program Files/…) outside the Bazel
sandbox, so install_ssldirs writes there and leaves the sandbox copy
empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows
FIPS (the FIPS provider comes from @openssl_fips). The fix drops these
from out_data_dirs on Windows FIPS via the existing fips_windows
config_setting.

### Describe how you validated your changes
Reproduced both failures locally on a Windows VM with rules_python 1.9.0,
confirmed both fixes resolve them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix Windows breakages in compliance/package_licenses.bzl and
deps/openssl.BUILD.bazel exposed by the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

The openssl FIPS build on Windows declares out_data_dirs = ["ssl",
"lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir
is set to an absolute path (C:/Program Files/…) outside the Bazel
sandbox, so install_ssldirs writes there and leaves the sandbox copy
empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows
FIPS (the FIPS provider comes from @openssl_fips). The fix drops these
from out_data_dirs on Windows FIPS via the existing fips_windows
config_setting.

### Describe how you validated your changes
Reproduced both failures locally on a Windows VM with rules_python 1.9.0,
confirmed both fixes resolve them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Mar 20, 2026
### What does this PR do?
Bump rules_python from 1.8.5 to 1.9.0 in isolation, as a prerequisite
for bumping rules_foreign_cc to a commit that requires its transitive
deps to be up-to-date.

Fix Windows breakages in compliance/package_licenses.bzl and
deps/openssl.BUILD.bazel exposed by the bump.

### Motivation
rules_python 1.9.0 enables runfiles on Windows for py_binary
(bazel-contrib/rules_python#3610). Where 1.8.5 pointed RUNFILES_DIR at
the build-time runfiles directory, 1.9.0 makes bazel run copy the
runfiles into a fresh temp directory. Bazel skips empty directories in
that copy.

package_licenses unconditionally included the offers_dir tree artifact
in pkg_files. ship_source_offer is not declared by any dep, so
offers_dir is always empty. On Linux and macOS bazel run uses symlinks,
so an empty tree is harmless. On Windows the directory is absent from
the temp copy, and pkg_install's copytree call fails with
FileNotFoundError. The fix excludes offers_dir on Windows. The problem
is still present in the latest rules_pkg main (bazelbuild/rules_pkg#1046
is unrelated).

The openssl FIPS build on Windows declares out_data_dirs = ["ssl",
"lib/ossl-modules", "lib/engines-3"]. ssl/ is empty because --openssldir
is set to an absolute path (C:/Program Files/…) outside the Bazel
sandbox, so install_ssldirs writes there and leaves the sandbox copy
empty. lib/ossl-modules/ and lib/engines-3/ are not installed on Windows
FIPS (the FIPS provider comes from @openssl_fips). The fix drops these
from out_data_dirs on Windows FIPS via the existing fips_windows
config_setting.

### Describe how you validated your changes
Reproduced both failures locally on a Windows VM with rules_python 1.9.0,
confirmed both fixes resolve them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d bot pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 23, 2026
… misc files packaging (#47807)

### What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
* The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
  * If a version is provided, symlinks for that version will be created automatically
  * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
* A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install

### Motivation

Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes.

At the end of the day, we want to:
* be able to automatically install all libraries and all their associated files
* automatically patch the rpath when installing
* automatically provide symlinks when installing
* not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

### Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

### Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
StephenWakely pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 27, 2026
### What does this PR do?
Bump `rules_pkg` to current `main`.

### Motivation
Pick up:
- bazelbuild/rules_pkg#1021: a contribution of ours
- bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it
- bazelbuild/rules_pkg#1035: affects reproducibiliy of builds
- bazelbuild/rules_pkg#1044
- bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette)
- bazelbuild/rules_pkg#1047: we might need this feature

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
StephenWakely pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 27, 2026
… misc files packaging (#47807)

### What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
* The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
  * If a version is provided, symlinks for that version will be created automatically
  * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
* A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install

### Motivation

Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes.

At the end of the day, we want to:
* be able to automatically install all libraries and all their associated files
* automatically patch the rpath when installing
* automatically provide symlinks when installing
* not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

### Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

### Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
StephenWakely pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 27, 2026
### What does this PR do?
Bump `rules_pkg` to current `main`.

### Motivation
Pick up:
- bazelbuild/rules_pkg#1021: a contribution of ours
- bazelbuild/rules_pkg#1024: RPM build errors would go unnoticed without it
- bazelbuild/rules_pkg#1035: affects reproducibiliy of builds
- bazelbuild/rules_pkg#1044
- bazelbuild/rules_pkg#1046: another contribution of ours (@chouquette)
- bazelbuild/rules_pkg#1047: we might need this feature

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
StephenWakely pushed a commit to DataDog/datadog-agent that referenced this pull request Mar 27, 2026
… misc files packaging (#47807)

### What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
* The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
  * If a version is provided, symlinks for that version will be created automatically
  * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
* A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install

### Motivation

Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes.

At the end of the day, we want to:
* be able to automatically install all libraries and all their associated files
* automatically patch the rpath when installing
* automatically provide symlinks when installing
* not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

### Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

### Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pkg_install fails if writing a symlink over an existing file

2 participants