Add new rules to automatically handle rpath patching, SO symlinking & misc files packaging#47807
Conversation
dc0b083 to
b09071f
Compare
Files inventory check summaryFile checks results against ancestor 178cf93a: Results for datadog-agent_7.78.0~devel.git.793.583927f.pipeline.103723087-1_amd64.deb:No change detected |
Static quality checks✅ Please find below the results from static quality gates 31 successful checks with minimal change (< 2 KiB)
On-wire sizes (compressed)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 1086150 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +4.09 | [+1.02, +7.17] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +4.09 | [+1.02, +7.17] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.66 | [+0.42, +0.89] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | +0.38 | [+0.24, +0.52] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.32 | [+0.26, +0.37] | 1 | Logs bounds checks dashboard |
| ➖ | docker_containers_memory | memory utilization | +0.15 | [+0.08, +0.22] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | +0.09 | [-1.41, +1.60] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_logs | memory utilization | +0.09 | [+0.02, +0.16] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.04 | [-0.35, +0.43] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | +0.02 | [-0.18, +0.21] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.19, +0.21] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.01 | [-0.08, +0.11] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | +0.01 | [-0.05, +0.07] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.01 | [-0.10, +0.09] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.02 | [-0.52, +0.48] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | -0.02 | [-0.12, +0.07] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | -0.04 | [-0.19, +0.11] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.06 | [-0.49, +0.36] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | -0.07 | [-0.24, +0.10] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.16 | [-0.30, -0.02] | 1 | Logs |
| ➖ | file_tree | memory utilization | -0.19 | [-0.25, -0.12] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.27 | [-0.31, -0.24] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics | memory utilization | -0.28 | [-0.46, -0.09] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.35 | [-0.58, -0.13] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 663 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 279.62MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 705 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.19GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.23GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.22GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 = 3 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 174.39MiB ≤ 175MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 2 ≤ 3 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 497.94MiB ≤ 550MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 202.90MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 357.70 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 409.83MiB ≤ 475MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
| "srcs": attr.label_list( | ||
| aspects = [_collect_dd_packaging_aspect], | ||
| ), | ||
| }, |
There was a problem hiding this comment.
provides = [PackageFilesInfo],
You should always list the explicit providers.
There was a problem hiding this comment.
done (with CcInfo and CcSharedLibraryInfo which are the expected provider for the srcs here)
There was a problem hiding this comment.
I meant something different.
This rule returns a PackageFilegroupInfo, so you should add provides = [PackageFilegroupInfo], after the attributes.
02d201b to
eed6e3a
Compare
There was a problem hiding this comment.
💡 Codex Review
datadog-agent/deps/openscap/BUILD.bazel
Lines 28 to 33 in ddeaba0
In the Bazel Linux packaging path I checked packages/agent/linux/BUILD.bazel:36-44, and transitive_deps still packages //deps/openscap:all_files. Removing @rpm//:all_files from that filegroup means those builds no longer ship librpm/librpmio, because the new dd_collect_dependencies logic only exists in @openscap//:install (deps/openscap/openscap.BUILD.bazel:663-676) and is never consulted by //deps/openscap:all_files. Any agent package assembled from packages/agent/linux will therefore miss the rpm shared libraries that openscap links against at runtime.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| pkg_files( | ||
| name = "lib_files", | ||
| srcs = [":z"], | ||
| ) |
There was a problem hiding this comment.
Preserve the versioned zlib symlink layout in @zlib//:install
lib_files now packages :z directly, so the standalone @zlib//:install target stops emitting the so_symlink chain that used to provide the versioned libz paths. I checked omnibus/config/software/openssl3.rb:32-33, and non-Windows OpenSSL builds still install zlib through that target, so any packaging or runtime lookup that expects the previous libz.so.1 / libz.1.dylib names will no longer find them even though the new dd_cc_packaged path still creates those links for transitive packaging.
Useful? React with 👍 / 👎.
481915b to
691fab1
Compare
|
This looks pretty good. Here is my test. It shows exactly the behavior we want. That the rebuild with a new --//:install_dir does not invalidate the build cache. My recommendation:
Then we can experiment with styles of making the dynamic_deps names correct - and fixing the duplicate installation of libz. that CI is breaking on. |
aiuto
left a comment
There was a problem hiding this comment.
Mostly great.
- Let's split it to be rules first, fixing the analysis test problem on darwin
- Then we can do a smaller review of changing all the deps.
You can probably have claude write that entire PR.
| "srcs": attr.label_list( | ||
| aspects = [_collect_dd_packaging_aspect], | ||
| ), | ||
| }, |
There was a problem hiding this comment.
I meant something different.
This rule returns a PackageFilegroupInfo, so you should add provides = [PackageFilegroupInfo], after the attributes.
| "srcs": attr.label_list( | ||
| doc = "Top-level CC targets which dependency graph should be searched for DdPackagingInfo.", | ||
| aspects = [_collect_dd_packaging_aspect], | ||
| providers = [[CcInfo], [CcSharedLibraryInfo]], |
There was a problem hiding this comment.
I don't think this is needed, or wanted.
We should eventually be able to colllect all sorts of things, even if they are not packaged.
For example, if you depend on a Go binary, drag along the configuration file for it.
There was a problem hiding this comment.
I guess that's fair, although I mostly meant for this rule to wrap cc_shared_library and cc_binary hence the name, but there's nothing that prevents it from being more language agnostic
| - input: _dd_cc_packaged_rule -> cc_shared_library edges (bridges a | ||
| packaged target back to its underlying cc_shared_library) | ||
| """, | ||
| attr_aspects = ["dynamic_deps", "input"], |
There was a problem hiding this comment.
We can totally expand this to add 'deps' one day.
But that is not essential now.
There was a problem hiding this comment.
True, I don't think we currently have statically linked libraries that also need to ship something with the agent (I'd argue that we shouldn't ship the headers if we statically link), but as you said, we can reconvene when needed
|
|
||
| def _test_packaged_forwards_unpatched_so_impl(env, target): | ||
| outputs = _outputs_of(env, target) | ||
| outputs.contains_predicate(matching.file_path_matches("*.so")) |
There was a problem hiding this comment.
Save yourself some debug time. Alex turned on test //... for macos, so this now fails on CI
-outputs.contains_predicate(matching.file_path_matches("*.so"))
+ outputs.contains_predicate(matching.any(
+ matching.file_path_matches("*.dll"),
+ matching.file_path_matches("*.dylib"),
+ matching.file_path_matches("*.so"),
+ ))
deps/zlib.BUILD.bazel
Outdated
| ) | ||
|
|
||
| dd_cc_packaged( | ||
| name = "z_so", |
There was a problem hiding this comment.
I think the migration process will be easier if
you rename the original ":z" to ":_z",
and then have this rule's name be "z".
That means almost all the existing dynamic_deps list remain the same.
But I don't consider that a blocker.
There was a problem hiding this comment.
good point, I'll make that change once I start tackling the PR to migrate individual deps
aaa745e to
426eee6
Compare
It's already indirectly exposed through the installed_files
it's an implementation detail for this package, no need to make it public
full disclosure: this was 100% generated by claude and reviewed afterward
it's unclear if we really need those until we start shipping that code outside of the agent. In any case, this can easily be regenerated if they are deemed to be useful
1bdf100 to
583927f
Compare
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
… misc files packaging (#47807) ### What does this PR do? This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries. * The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects. * If a version is provided, symlinks for that version will be created automatically * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time * A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install ### Motivation Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes. At the end of the day, we want to: * be able to automatically install all libraries and all their associated files * automatically patch the rpath when installing * automatically provide symlinks when installing * not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on. ### Describe how you validated your changes Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally ### Additional Notes This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this. RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward. All namings are debatable, no doc was provided until the API is settled. This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored) Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
… misc files packaging (#47807) ### What does this PR do? This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries. * The `dd_cc_packaged` macro serves as a wrapper on top of `CcInfo` or `CcSharedLibrary`. It can be used transparently just like a `cc_binary` or `cc_shared_library` executable, but also takes an `installed_files` attribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects. * If a version is provided, symlinks for that version will be created automatically * The macro automatically appends the rpath patched version of the binary to the list of files to install at packaging time * A `dd_collect_dependencies` rule that leverages a new `_collect_dd_packaging_aspect` aspect. It automatically walks the build tree to find all binaries provided by `dd_cc_packaged` and includes their `installed_files` to the list of files to install ### Motivation Getting rid of the manual `rewrite_rpath` invocations that are left in the omnibus recipes. At the end of the day, we want to: * be able to automatically install all libraries and all their associated files * automatically patch the rpath when installing * automatically provide symlinks when installing * not having to care about what dependencies should be pulled in with a tool that we want to install. If we need openscap to be installed, we should include all its dependencies, their dependencies' dependencies, and so on. ### Describe how you validated your changes Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally ### Additional Notes This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this. RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from `openscap.rb` and check that we don't miss any file afterward. All namings are debatable, no doc was provided until the API is settled. This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored) Co-authored-by: hugo.beauzee <hugo.beauzee@datadoghq.com>
What does this PR do?
This PR adds a few new rules & aspect in order to simplify our dependencies management, and drop the need to manually patch rpath for our executables & shared libraries.
dd_cc_packagedmacro serves as a wrapper on top ofCcInfoorCcSharedLibrary. It can be used transparently just like acc_binaryorcc_shared_libraryexecutable, but also takes aninstalled_filesattribute, which can bundle a set of files along with the binary. At installation time, these files will be installed along with the executable and/or shared objects.dd_collect_dependenciesrule that leverages a new_collect_dd_packaging_aspectaspect. It automatically walks the build tree to find all binaries provided bydd_cc_packagedand includes theirinstalled_filesto the list of files to installMotivation
Getting rid of the manual
rewrite_rpathinvocations that are left in the omnibus recipes.At the end of the day, we want to:
Describe how you validated your changes
Mostly to be done through the CI and file inventory check, but also by manually building & installing openscap locally
Additional Notes
This is only at the POC level. It only handles zlib so prove that we can still build with a wrapped object, and since zlib is the most commonly used lib, it's a good candidate for this.
RPM is used as the example for automatic installation as it's only used in openscap, so it's simple to just remove it from
openscap.rband check that we don't miss any file afterward.All namings are debatable, no doc was provided until the API is settled.
This needs bazelbuild/rules_pkg#1046 to be merged (or the patch to be vendored)