Skip to content

Add new rules to automatically handle rpath patching, SO symlinking & misc files packaging#47807

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 25 commits intomainfrom
chouquette/bazel/automatic_deps_rpath/1
Mar 23, 2026
Merged

Add new rules to automatically handle rpath patching, SO symlinking & misc files packaging#47807
gh-worker-dd-mergequeue-cf854d[bot] merged 25 commits intomainfrom
chouquette/bazel/automatic_deps_rpath/1

Conversation

@chouquette
Copy link
Copy Markdown
Contributor

@chouquette chouquette commented Mar 13, 2026

What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.

  • The dd_cc_packaged macro serves as a wrapper on top of CcInfo or CcSharedLibrary. It can be used transparently just like a cc_binary or cc_shared_library executable, but also takes an installed_files attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
    • If a version is provided, symlinks for that version will be created automatically
    • The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
  • A dd_collect_dependencies rule that leverages a new _collect_dd_packaging_aspect aspect. It automatically walks the build tree to find all binaries provided by dd_cc_packaged and includes their installed_files to the list of files to install

Motivation

Getting rid of the manual rewrite_rpath invocations that are left in the omnibus recipes.

At the end of the day, we want to:

  • be able to automatically install all libraries and all their associated files
  • automatically patch the rpath when installing
  • automatically provide symlinks when installing
  • not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from openscap.rb and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

@github-actions github-actions bot added the long review PR is complex, plan time to review it label Mar 13, 2026
@chouquette chouquette added changelog/no-changelog No changelog entry needed qa/no-code-change No code change in Agent code requiring validation labels Mar 13, 2026
@chouquette chouquette force-pushed the chouquette/bazel/automatic_deps_rpath/1 branch 2 times, most recently from dc0b083 to b09071f Compare March 16, 2026 11:21
@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr bot commented Mar 16, 2026

Files inventory check summary

File checks results against ancestor 178cf93a:

Results for datadog-agent_7.78.0~devel.git.793.583927f.pipeline.103723087-1_amd64.deb:

No change detected

@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr bot commented Mar 16, 2026

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor d9be88a
📊 Static Quality Gates Dashboard
🔗 SQG Job

31 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
agent_deb_amd64 750.700 MiB
agent_deb_amd64_fips 707.632 MiB
agent_heroku_amd64 313.003 MiB
agent_msi 604.238 MiB
agent_rpm_amd64 750.684 MiB
agent_rpm_amd64_fips 707.616 MiB
agent_rpm_arm64 729.015 MiB
agent_rpm_arm64_fips 689.007 MiB
agent_suse_amd64 750.684 MiB
agent_suse_amd64_fips 707.616 MiB
agent_suse_arm64 729.015 MiB
agent_suse_arm64_fips 689.007 MiB
docker_agent_amd64 811.019 MiB
docker_agent_arm64 814.164 MiB
docker_agent_jmx_amd64 1001.935 MiB
docker_agent_jmx_arm64 993.858 MiB
docker_cluster_agent_amd64 205.292 MiB
docker_cluster_agent_arm64 219.614 MiB
docker_cws_instrumentation_amd64 7.142 MiB
docker_cws_instrumentation_arm64 6.689 MiB
docker_dogstatsd_amd64 39.215 MiB
docker_dogstatsd_arm64 37.445 MiB
dogstatsd_deb_amd64 29.855 MiB
dogstatsd_deb_arm64 28.007 MiB
dogstatsd_rpm_amd64 29.855 MiB
dogstatsd_suse_amd64 29.855 MiB
iot_agent_deb_amd64 43.218 MiB
iot_agent_deb_arm64 40.273 MiB
iot_agent_deb_armhf 41.017 MiB
iot_agent_rpm_amd64 43.219 MiB
iot_agent_suse_amd64 43.219 MiB
On-wire sizes (compressed)
Quality gate Change Size (prev → curr → max)
agent_deb_amd64 +39.99 KiB (0.02% increase) 174.602 → 174.641 → 178.360
agent_deb_amd64_fips +2.86 KiB (0.00% increase) 165.198 → 165.201 → 172.790
agent_heroku_amd64 neutral 74.943 MiB → 79.970
agent_msi -12.0 KiB (0.01% reduction) 138.250 → 138.238 → 146.220
agent_rpm_amd64 -14.46 KiB (0.01% reduction) 177.533 → 177.519 → 181.830
agent_rpm_amd64_fips -7.55 KiB (0.00% reduction) 167.282 → 167.275 → 173.370
agent_rpm_arm64 neutral 159.517 MiB → 163.060
agent_rpm_arm64_fips -34.26 KiB (0.02% reduction) 151.274 → 151.241 → 156.170
agent_suse_amd64 -14.46 KiB (0.01% reduction) 177.533 → 177.519 → 181.830
agent_suse_amd64_fips -7.55 KiB (0.00% reduction) 167.282 → 167.275 → 173.370
agent_suse_arm64 neutral 159.517 MiB → 163.060
agent_suse_arm64_fips -34.26 KiB (0.02% reduction) 151.274 → 151.241 → 156.170
docker_agent_amd64 neutral 267.885 MiB → 272.480
docker_agent_arm64 -7.49 KiB (0.00% reduction) 255.092 → 255.084 → 261.060
docker_agent_jmx_amd64 neutral 336.532 MiB → 341.100
docker_agent_jmx_arm64 -8.6 KiB (0.00% reduction) 319.727 → 319.719 → 325.620
docker_cluster_agent_amd64 neutral 71.921 MiB → 72.920
docker_cluster_agent_arm64 neutral 67.506 MiB → 68.220
docker_cws_instrumentation_amd64 neutral 2.999 MiB → 3.330
docker_cws_instrumentation_arm64 neutral 2.729 MiB → 3.090
docker_dogstatsd_amd64 neutral 15.161 MiB → 15.820
docker_dogstatsd_arm64 neutral 14.480 MiB → 14.830
dogstatsd_deb_amd64 neutral 7.888 MiB → 8.790
dogstatsd_deb_arm64 neutral 6.774 MiB → 7.710
dogstatsd_rpm_amd64 neutral 7.898 MiB → 8.800
dogstatsd_suse_amd64 neutral 7.898 MiB → 8.800
iot_agent_deb_amd64 neutral 11.385 MiB → 12.040
iot_agent_deb_arm64 neutral 9.694 MiB → 10.450
iot_agent_deb_armhf neutral 9.932 MiB → 10.620
iot_agent_rpm_amd64 neutral 11.400 MiB → 12.060
iot_agent_suse_amd64 neutral 11.400 MiB → 12.060

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Mar 16, 2026

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: ec7bdeb5-146d-4f96-b2bb-bd8d157dadaf

Baseline: 1086150
Comparison: 2608109
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization +4.09 [+1.02, +7.17] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization +4.09 [+1.02, +7.17] 1 Logs
quality_gate_metrics_logs memory utilization +0.66 [+0.42, +0.89] 1 Logs bounds checks dashboard
ddot_metrics_sum_cumulative memory utilization +0.38 [+0.24, +0.52] 1 Logs
quality_gate_idle memory utilization +0.32 [+0.26, +0.37] 1 Logs bounds checks dashboard
docker_containers_memory memory utilization +0.15 [+0.08, +0.22] 1 Logs
quality_gate_logs % cpu utilization +0.09 [-1.41, +1.60] 1 Logs bounds checks dashboard
ddot_logs memory utilization +0.09 [+0.02, +0.16] 1 Logs
file_to_blackhole_500ms_latency egress throughput +0.04 [-0.35, +0.43] 1 Logs
uds_dogstatsd_to_api_v3 ingress throughput +0.02 [-0.18, +0.21] 1 Logs
uds_dogstatsd_to_api ingress throughput +0.01 [-0.19, +0.21] 1 Logs
file_to_blackhole_100ms_latency egress throughput +0.01 [-0.08, +0.11] 1 Logs
uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization +0.01 [-0.05, +0.07] 1 Logs
tcp_dd_logs_filter_exclude ingress throughput -0.01 [-0.10, +0.09] 1 Logs
file_to_blackhole_0ms_latency egress throughput -0.02 [-0.52, +0.48] 1 Logs
otlp_ingest_logs memory utilization -0.02 [-0.12, +0.07] 1 Logs
otlp_ingest_metrics memory utilization -0.04 [-0.19, +0.11] 1 Logs
file_to_blackhole_1000ms_latency egress throughput -0.06 [-0.49, +0.36] 1 Logs
ddot_metrics_sum_delta memory utilization -0.07 [-0.24, +0.10] 1 Logs
tcp_syslog_to_blackhole ingress throughput -0.16 [-0.30, -0.02] 1 Logs
file_tree memory utilization -0.19 [-0.25, -0.12] 1 Logs
quality_gate_idle_all_features memory utilization -0.27 [-0.31, -0.24] 1 Logs bounds checks dashboard
ddot_metrics memory utilization -0.28 [-0.46, -0.09] 1 Logs
ddot_metrics_sum_cumulativetodelta_exporter memory utilization -0.35 [-0.58, -0.13] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
docker_containers_cpu simple_check_run 10/10 663 ≥ 26
docker_containers_memory memory_usage 10/10 279.62MiB ≤ 370MiB
docker_containers_memory simple_check_run 10/10 705 ≥ 26
file_to_blackhole_0ms_latency memory_usage 10/10 0.19GiB ≤ 1.20GiB
file_to_blackhole_0ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_1000ms_latency memory_usage 10/10 0.23GiB ≤ 1.20GiB
file_to_blackhole_1000ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_100ms_latency memory_usage 10/10 0.20GiB ≤ 1.20GiB
file_to_blackhole_100ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_500ms_latency memory_usage 10/10 0.22GiB ≤ 1.20GiB
file_to_blackhole_500ms_latency missed_bytes 10/10 0B = 0B
quality_gate_idle intake_connections 10/10 3 = 3 bounds checks dashboard
quality_gate_idle memory_usage 10/10 174.39MiB ≤ 175MiB bounds checks dashboard
quality_gate_idle_all_features intake_connections 10/10 2 ≤ 3 bounds checks dashboard
quality_gate_idle_all_features memory_usage 10/10 497.94MiB ≤ 550MiB bounds checks dashboard
quality_gate_logs intake_connections 10/10 4 ≤ 6 bounds checks dashboard
quality_gate_logs memory_usage 10/10 202.90MiB ≤ 220MiB bounds checks dashboard
quality_gate_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
quality_gate_metrics_logs cpu_usage 10/10 357.70 ≤ 2000 bounds checks dashboard
quality_gate_metrics_logs intake_connections 10/10 3 ≤ 6 bounds checks dashboard
quality_gate_metrics_logs memory_usage 10/10 409.83MiB ≤ 475MiB bounds checks dashboard
quality_gate_metrics_logs missed_bytes 10/10 0B = 0B bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

Passed. All Quality Gates passed.

  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.

"srcs": attr.label_list(
aspects = [_collect_dd_packaging_aspect],
),
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provides = [PackageFilesInfo],

You should always list the explicit providers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done (with CcInfo and CcSharedLibraryInfo which are the expected provider for the srcs here)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant something different.
This rule returns a PackageFilegroupInfo, so you should add provides = [PackageFilegroupInfo], after the attributes.

@chouquette chouquette force-pushed the chouquette/bazel/automatic_deps_rpath/1 branch 4 times, most recently from 02d201b to eed6e3a Compare March 18, 2026 11:26
@chouquette chouquette marked this pull request as ready for review March 18, 2026 11:31
@chouquette chouquette requested review from a team as code owners March 18, 2026 11:31
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

srcs = [
":all_embedded",
"@acl//:all_files",
"@attr//:all_files",
"@bzip2//:all_files",
],

P1 Badge Keep RPM artifacts in //deps/openscap:all_files

In the Bazel Linux packaging path I checked packages/agent/linux/BUILD.bazel:36-44, and transitive_deps still packages //deps/openscap:all_files. Removing @rpm//:all_files from that filegroup means those builds no longer ship librpm/librpmio, because the new dd_collect_dependencies logic only exists in @openscap//:install (deps/openscap/openscap.BUILD.bazel:663-676) and is never consulted by //deps/openscap:all_files. Any agent package assembled from packages/agent/linux will therefore miss the rpm shared libraries that openscap links against at runtime.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +142 to +145
pkg_files(
name = "lib_files",
srcs = [":z"],
)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the versioned zlib symlink layout in @zlib//:install

lib_files now packages :z directly, so the standalone @zlib//:install target stops emitting the so_symlink chain that used to provide the versioned libz paths. I checked omnibus/config/software/openssl3.rb:32-33, and non-Windows OpenSSL builds still install zlib through that target, so any packaging or runtime lookup that expects the previous libz.so.1 / libz.1.dylib names will no longer find them even though the new dd_cc_packaged path still creates those links for transitive packaging.

Useful? React with 👍 / 👎.

@chouquette chouquette force-pushed the chouquette/bazel/automatic_deps_rpath/1 branch 2 times, most recently from 481915b to 691fab1 Compare March 18, 2026 15:04
@aiuto
Copy link
Copy Markdown
Contributor

aiuto commented Mar 18, 2026

This looks pretty good. Here is my test.

bazel build \
  //packages/agent/linux:whole_distro_tar \
  //packages/agent/linux:collected_oscap_deps_embedded @openscap//:oscap @zlib//:z
ls -l $(find bazel-bin/. -name '*.so' | sort) >/tmp/stuff.default
bazel build --//:install_dir=/tmp/FIXED \
   //packages/agent/linux:collected_oscap_deps_embedded @openscap//:oscap @zlib//:z
ls -l $(find bazel-bin/. -name '*.so' | sort) >/tmp/stuff.fixed
echo This should show minimal diffs, indicating that everything except the librares is cached.
diff /tmp/stuff.default /tmp/stuff.fixed
patchelf --print-rpath bazel-out/aarch64-fastbuild/bin/external/+_repo_rules+rpm/librpm.so
patchelf --print-rpath bazel-out/aarch64-fastbuild/bin/external/+_repo_rules+rpm/patched/librpm.so

It shows exactly the behavior we want. That the rebuild with a new --//:install_dir does not invalidate the build cache.

My recommendation:

  • If we don't need collect_cc_dynamic_deps_bzl, delete it in a distinct PR first, or last. But removing it is independent of the new work, so it belongs in a new PR.
  • Split out a PR to add bazel/rules/dd_packaging/... And the bzl_library stanzas for the otehrs if you need them. There should be nothing blocking that. I'll approve that in an instant.

Then we can experiment with styles of making the dynamic_deps names correct - and fixing the duplicate installation of libz. that CI is breaking on.
Finally, one PR that does all the deps.

Copy link
Copy Markdown
Contributor

@aiuto aiuto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly great.

  • Let's split it to be rules first, fixing the analysis test problem on darwin
  • Then we can do a smaller review of changing all the deps.
    You can probably have claude write that entire PR.

"srcs": attr.label_list(
aspects = [_collect_dd_packaging_aspect],
),
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant something different.
This rule returns a PackageFilegroupInfo, so you should add provides = [PackageFilegroupInfo], after the attributes.

"srcs": attr.label_list(
doc = "Top-level CC targets which dependency graph should be searched for DdPackagingInfo.",
aspects = [_collect_dd_packaging_aspect],
providers = [[CcInfo], [CcSharedLibraryInfo]],
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is needed, or wanted.
We should eventually be able to colllect all sorts of things, even if they are not packaged.
For example, if you depend on a Go binary, drag along the configuration file for it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that's fair, although I mostly meant for this rule to wrap cc_shared_library and cc_binary hence the name, but there's nothing that prevents it from being more language agnostic

- input: _dd_cc_packaged_rule -> cc_shared_library edges (bridges a
packaged target back to its underlying cc_shared_library)
""",
attr_aspects = ["dynamic_deps", "input"],
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can totally expand this to add 'deps' one day.

But that is not essential now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I don't think we currently have statically linked libraries that also need to ship something with the agent (I'd argue that we shouldn't ship the headers if we statically link), but as you said, we can reconvene when needed


def _test_packaged_forwards_unpatched_so_impl(env, target):
outputs = _outputs_of(env, target)
outputs.contains_predicate(matching.file_path_matches("*.so"))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Save yourself some debug time. Alex turned on test //... for macos, so this now fails on CI

-outputs.contains_predicate(matching.file_path_matches("*.so"))
+    outputs.contains_predicate(matching.any(
+        matching.file_path_matches("*.dll"),
+        matching.file_path_matches("*.dylib"),
+        matching.file_path_matches("*.so"),
+    ))

)

dd_cc_packaged(
name = "z_so",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the migration process will be easier if
you rename the original ":z" to ":_z",
and then have this rule's name be "z".

That means almost all the existing dynamic_deps list remain the same.

But I don't consider that a blocker.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I'll make that change once I start tackling the PR to migrate individual deps

@chouquette chouquette force-pushed the chouquette/bazel/automatic_deps_rpath/1 branch 2 times, most recently from aaa745e to 426eee6 Compare March 19, 2026 09:45
@chouquette chouquette force-pushed the chouquette/bazel/automatic_deps_rpath/1 branch from 1bdf100 to 583927f Compare March 20, 2026 13:43
@chouquette
Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Mar 23, 2026

View all feedbacks in Devflow UI.

2026-03-23 07:56:00 UTC ℹ️ Start processing command /merge


2026-03-23 07:56:05 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 2h (p90).


2026-03-23 08:35:45 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot merged commit 2608109 into main Mar 23, 2026
259 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot deleted the chouquette/bazel/automatic_deps_rpath/1 branch March 23, 2026 08:35
@github-actions github-actions bot added this to the 7.79.0 milestone Mar 23, 2026
StephenWakely pushed a commit that referenced this pull request Mar 27, 2026
… misc files packaging (#47807)

### What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
* The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
  * If a version is provided, symlinks for that version will be created automatically
  * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
* A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install

### Motivation

Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes.

At the end of the day, we want to:
* be able to automatically install all libraries and all their associated files
* automatically patch the rpath when installing
* automatically provide symlinks when installing
* not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

### Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

### Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
StephenWakely pushed a commit that referenced this pull request Mar 27, 2026
… misc files packaging (#47807)

### What does this PR do?

This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
* The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.
  * If a version is provided, symlinks for that version will be created automatically
  * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time
* A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install

### Motivation

Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes.

At the end of the day, we want to:
* be able to automatically install all libraries and all their associated files
* automatically patch the rpath when installing
* automatically provide symlinks when installing
* not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on.

### Describe how you validated your changes

Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally

### Additional Notes

This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward.

All namings are debatable, no doc was provided until the API is settled.

This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)

Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog No changelog entry needed internal Identify a non-fork PR long review PR is complex, plan time to review it qa/no-code-change No code change in Agent code requiring validation team/agent-build team/agent-cspm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants