Skip to content

fix(ci): Re-add custom builds workflow. Fixes integration test build error. #17024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
neuronull merged 7 commits into from
Apr 3, 2023
Merged

fix(ci): Re-add custom builds workflow. Fixes integration test build error. #17024

neuronull merged 7 commits into from
Apr 3, 2023

Conversation

@neuronull
Copy link
Contributor

@neuronull neuronull commented Mar 31, 2023 •
edited
Loading

Revival of #16281

Fixes the issue with integration tests not building- needed to install git dependency for the container.

The k8s e2e tests needed to have the version exported for the containerized build.

From the commit history you can see there is no other functional change from the original PR, thus I am not planning to set the trigger to PR and run the custom builds workflow, because that was already proved in the original PR.

See commit 07aafb6
and baf1b9e
, for the fix.

@neuronull neuronull self-assigned this Mar 31, 2023
@netlify
Copy link

netlify bot commented Mar 31, 2023 •
edited
Loading

✅ Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit baf1b9e
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/64275024af9c370008b94fdd

@github-actions github-actions bot added domain: ci Anything related to Vector's CI environment domain: releasing Anything related to releasing Vector labels Mar 31, 2023
@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Mar 31, 2023 •
edited
Loading

Datadog Report

Branch report: neuronull/custom_builds_int_tests_fix
Commit report: 1419b11

✅ vector: 0 Failed, 0 New Flaky, 2130 Passed, 0 Skipped, 8m 31.01s Wall Time

jszwedko
jszwedko approved these changes Mar 31, 2023
View reviewed changes
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

@github-actions
Copy link

github-actions bot commented Mar 31, 2023

Regression Detector Results

Run ID: 8ef35ed4-02e8-4e73-bf36-3d4f33cc60a0
Baseline: 00c0316
Comparison: 07aafb6
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
file_to_blackhole egress throughput 759.91KiB/CPU-s 12.60 100.00% 5.89MiB/CPU-s 4.51MiB/CPU-s 123.61KiB/CPU-s 0.0 0.765514 6.63MiB/CPU-s 4.57MiB/CPU-s 131.91KiB/CPU-s 0.0 0.688708 True True
datadog_agent_remap_datadog_logs_acks ingress throughput 822.99KiB/CPU-s 2.56 100.00% 31.4MiB/CPU-s 1.36MiB/CPU-s 17.17KiB/CPU-s 0.0 0.043397 32.21MiB/CPU-s 1.26MiB/CPU-s 15.83KiB/CPU-s 0.0 0.039017 False False
splunk_hec_route_s3 ingress throughput 218.86KiB/CPU-s 1.88 100.00% 11.34MiB/CPU-s 674.59KiB/CPU-s 8.3KiB/CPU-s 0.0 0.058083 11.55MiB/CPU-s 558.32KiB/CPU-s 6.87KiB/CPU-s 0.0 0.047183 False False
http_to_http_acks ingress throughput 88.69KiB/CPU-s 1.66 93.50% 5.21MiB/CPU-s 2.67MiB/CPU-s 33.63KiB/CPU-s 0.0 0.512581 5.29MiB/CPU-s 2.72MiB/CPU-s 34.32KiB/CPU-s 0.0 0.51451 True False
datadog_agent_remap_datadog_logs ingress throughput 364.04KiB/CPU-s 1.08 100.00% 33.01MiB/CPU-s 792.28KiB/CPU-s 9.75KiB/CPU-s 0.0 0.023438 33.36MiB/CPU-s 986.68KiB/CPU-s 12.14KiB/CPU-s 0.0 0.028877 False False
datadog_agent_remap_blackhole ingress throughput 191.48KiB/CPU-s 0.61 100.00% 30.5MiB/CPU-s 1.18MiB/CPU-s 14.89KiB/CPU-s 0.0 0.038743 30.68MiB/CPU-s 1.41MiB/CPU-s 17.7KiB/CPU-s 0.0 0.04579 False False
datadog_agent_remap_blackhole_acks ingress throughput 116.18KiB/CPU-s 0.37 100.00% 31.08MiB/CPU-s 1015.96KiB/CPU-s 12.5KiB/CPU-s 0.0 0.031923 31.19MiB/CPU-s 813.69KiB/CPU-s 10.02KiB/CPU-s 0.0 0.025475 False False
syslog_humio_logs ingress throughput 7.95KiB/CPU-s 0.09 90.58% 8.9MiB/CPU-s 261.22KiB/CPU-s 3.21KiB/CPU-s 0.0 0.028666 8.91MiB/CPU-s 284.56KiB/CPU-s 3.5KiB/CPU-s 0.0 0.0312 False False
enterprise_http_to_http ingress throughput 3.8KiB/CPU-s 0.03 61.85% 13.62MiB/CPU-s 277.52KiB/CPU-s 3.41KiB/CPU-s 0.0 0.019901 13.62MiB/CPU-s 218.45KiB/CPU-s 2.69KiB/CPU-s 0.0 0.015661 False False
http_text_to_http_json ingress throughput 4.59KiB/CPU-s 0.02 30.79% 23.93MiB/CPU-s 522.07KiB/CPU-s 6.42KiB/CPU-s 0.0 0.021303 23.93MiB/CPU-s 784.69KiB/CPU-s 9.66KiB/CPU-s 0.0 0.032014 False False
http_to_http_json ingress throughput 1.18KiB/CPU-s 0.01 23.28% 13.62MiB/CPU-s 230.03KiB/CPU-s 2.83KiB/CPU-s 0.0 0.016492 13.62MiB/CPU-s 229.08KiB/CPU-s 2.82KiB/CPU-s 0.0 0.016423 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput 1.26KiB/CPU-s 0.01 22.07% 13.62MiB/CPU-s 261.17KiB/CPU-s 3.21KiB/CPU-s 0.0 0.01873 13.62MiB/CPU-s 253.71KiB/CPU-s 3.12KiB/CPU-s 0.0 0.018194 False False
fluent_elasticsearch ingress throughput 26.42B/CPU-s 0.00 3.95% 45.41MiB/CPU-s 30.43KiB/CPU-s 379.25B/CPU-s 0.0 0.000654 45.41MiB/CPU-s 30.1KiB/CPU-s 375.04B/CPU-s 0.0 0.000647 False False
splunk_hec_indexer_ack_blackhole ingress throughput -52.39B/CPU-s -0.00 0.90% 13.62MiB/CPU-s 260.19KiB/CPU-s 3.2KiB/CPU-s 0.0 0.018661 13.62MiB/CPU-s 262.27KiB/CPU-s 3.23KiB/CPU-s 0.0 0.01881 False False
http_to_http_noack ingress throughput -216.18B/CPU-s -0.00 2.74% 13.61MiB/CPU-s 349.93KiB/CPU-s 4.3KiB/CPU-s 0.0 0.025107 13.61MiB/CPU-s 357.32KiB/CPU-s 4.4KiB/CPU-s 0.0 0.025638 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 319.76B/CPU-s 0.00 4.00% 13.61MiB/CPU-s 355.43KiB/CPU-s 4.37KiB/CPU-s 0.0 0.025494 13.61MiB/CPU-s 361.1KiB/CPU-s 4.44KiB/CPU-s 0.0 0.025901 False False
otlp_http_to_blackhole ingress throughput -1006.07B/CPU-s -0.06 38.83% 1.55MiB/CPU-s 116.83KiB/CPU-s 1.44KiB/CPU-s 0.0 0.073679 1.55MiB/CPU-s 105.34KiB/CPU-s 1.3KiB/CPU-s 0.0 0.066472 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -10.01KiB/CPU-s -0.28 85.73% 3.47MiB/CPU-s 355.74KiB/CPU-s 4.38KiB/CPU-s 0.0 0.100183 3.46MiB/CPU-s 425.54KiB/CPU-s 5.24KiB/CPU-s 0.0 0.120177 True True
syslog_loki ingress throughput -37.96KiB/CPU-s -0.44 100.00% 8.42MiB/CPU-s 250.17KiB/CPU-s 3.08KiB/CPU-s 0.0 0.02902 8.38MiB/CPU-s 271.61KiB/CPU-s 3.34KiB/CPU-s 0.0 0.031646 False False
socket_to_socket_blackhole ingress throughput -68.13KiB/CPU-s -0.50 100.00% 13.27MiB/CPU-s 373.64KiB/CPU-s 4.6KiB/CPU-s 0.0 0.027494 13.2MiB/CPU-s 443.19KiB/CPU-s 5.45KiB/CPU-s 0.0 0.032776 False False
otlp_grpc_to_blackhole ingress throughput -13.47KiB/CPU-s -1.28 100.00% 1.03MiB/CPU-s 46.74KiB/CPU-s 588.96B/CPU-s 0.0 0.044394 1.01MiB/CPU-s 53.16KiB/CPU-s 669.87B/CPU-s 0.0 0.051147 False False
syslog_splunk_hec_logs ingress throughput -134.86KiB/CPU-s -1.46 100.00% 9.04MiB/CPU-s 234.17KiB/CPU-s 2.88KiB/CPU-s 0.0 0.025304 8.9MiB/CPU-s 192.79KiB/CPU-s 2.37KiB/CPU-s 0.0 0.021141 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -345.87KiB/CPU-s -3.58 100.00% 9.42MiB/CPU-s 239.29KiB/CPU-s 2.94KiB/CPU-s 0.0 0.024793 9.09MiB/CPU-s 286.46KiB/CPU-s 3.52KiB/CPU-s 0.0 0.030784 False False
syslog_log2metric_humio_metrics ingress throughput -240.05KiB/CPU-s -3.83 100.00% 6.13MiB/CPU-s 208.89KiB/CPU-s 2.57KiB/CPU-s 0.0 0.033298 5.89MiB/CPU-s 355.59KiB/CPU-s 4.38KiB/CPU-s 0.0 0.058937 False False

@github-actions
Copy link

github-actions bot commented Mar 31, 2023

Regression Detector Results

Run ID: ed0b99e1-1c3c-4309-8131-590561a2779e
Baseline: 00c0316
Comparison: b93b3de
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
otlp_grpc_to_blackhole ingress throughput 9.93KiB/CPU-s 0.95 100.00% 1.02MiB/CPU-s 48.7KiB/CPU-s 613.64B/CPU-s 0.0 0.046771 1.03MiB/CPU-s 45.64KiB/CPU-s 575.13B/CPU-s 0.0 0.043421 False False
datadog_agent_remap_datadog_logs ingress throughput 290.67KiB/CPU-s 0.88 100.00% 32.24MiB/CPU-s 1.12MiB/CPU-s 14.15KiB/CPU-s 0.0 0.034834 32.53MiB/CPU-s 1.28MiB/CPU-s 16.11KiB/CPU-s 0.0 0.039302 False False
file_to_blackhole egress throughput 44.82KiB/CPU-s 0.67 19.09% 6.55MiB/CPU-s 4.65MiB/CPU-s 131.53KiB/CPU-s 2.290076 0.709417 6.59MiB/CPU-s 4.51MiB/CPU-s 130.79KiB/CPU-s 0.0 0.684219 True True
http_text_to_http_json ingress throughput 127.62KiB/CPU-s 0.52 100.00% 23.89MiB/CPU-s 599.08KiB/CPU-s 7.37KiB/CPU-s 0.0 0.024485 24.02MiB/CPU-s 836.32KiB/CPU-s 10.29KiB/CPU-s 0.0 0.034003 False False
http_to_http_json ingress throughput 64.7KiB/CPU-s 0.47 100.00% 13.55MiB/CPU-s 335.61KiB/CPU-s 4.13KiB/CPU-s 0.0 0.024185 13.61MiB/CPU-s 243.42KiB/CPU-s 2.99KiB/CPU-s 0.0 0.01746 False False
datadog_agent_remap_blackhole ingress throughput 135.89KiB/CPU-s 0.44 100.00% 30.44MiB/CPU-s 1.03MiB/CPU-s 12.92KiB/CPU-s 0.0 0.033677 30.57MiB/CPU-s 1.35MiB/CPU-s 16.97KiB/CPU-s 0.0 0.044045 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 121.75KiB/CPU-s 0.36 100.00% 32.86MiB/CPU-s 1.04MiB/CPU-s 13.04KiB/CPU-s 0.0 0.031516 32.98MiB/CPU-s 961.2KiB/CPU-s 11.83KiB/CPU-s 0.0 0.028464 False False
syslog_regex_logs2metric_ddmetrics ingress throughput 8.58KiB/CPU-s 0.23 74.13% 3.61MiB/CPU-s 404.89KiB/CPU-s 4.98KiB/CPU-s 0.0 0.109462 3.62MiB/CPU-s 466.17KiB/CPU-s 5.74KiB/CPU-s 0.0 0.125739 True True
otlp_http_to_blackhole ingress throughput 1.54KiB/CPU-s 0.10 54.53% 1.53MiB/CPU-s 119.71KiB/CPU-s 1.47KiB/CPU-s 0.0 0.076499 1.53MiB/CPU-s 116.85KiB/CPU-s 1.44KiB/CPU-s 0.0 0.074598 False False
enterprise_http_to_http ingress throughput 3.86KiB/CPU-s 0.03 68.27% 13.62MiB/CPU-s 249.64KiB/CPU-s 3.07KiB/CPU-s 0.0 0.0179 13.62MiB/CPU-s 189.23KiB/CPU-s 2.33KiB/CPU-s 0.0 0.013564 False False
http_to_http_noack ingress throughput 191.7B/CPU-s 0.00 2.29% 13.61MiB/CPU-s 377.55KiB/CPU-s 4.64KiB/CPU-s 0.0 0.027093 13.61MiB/CPU-s 373.09KiB/CPU-s 4.59KiB/CPU-s 0.0 0.026773 False False
splunk_hec_indexer_ack_blackhole ingress throughput -1.1KiB/CPU-s -0.01 19.15% 13.62MiB/CPU-s 259.33KiB/CPU-s 3.19KiB/CPU-s 0.0 0.018599 13.61MiB/CPU-s 264.57KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018976 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -1.01KiB/CPU-s -0.01 13.45% 13.61MiB/CPU-s 334.6KiB/CPU-s 4.12KiB/CPU-s 0.0 0.023998 13.61MiB/CPU-s 349.52KiB/CPU-s 4.3KiB/CPU-s 0.0 0.02507 False False
fluent_elasticsearch ingress throughput -5.49KiB/CPU-s -0.01 87.81% 45.41MiB/CPU-s 29.7KiB/CPU-s 370.11B/CPU-s 0.0 0.000639 45.41MiB/CPU-s 290.35KiB/CPU-s 3.53KiB/CPU-s 0.0 0.006244 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -2.87KiB/CPU-s -0.02 47.66% 13.62MiB/CPU-s 247.35KiB/CPU-s 3.04KiB/CPU-s 0.0 0.017737 13.61MiB/CPU-s 269.41KiB/CPU-s 3.31KiB/CPU-s 0.0 0.019323 False False
socket_to_socket_blackhole ingress throughput -21.47KiB/CPU-s -0.16 99.97% 13.27MiB/CPU-s 275.92KiB/CPU-s 3.4KiB/CPU-s 0.0 0.0203 13.25MiB/CPU-s 392.99KiB/CPU-s 4.84KiB/CPU-s 0.0 0.028958 False False
http_to_http_acks ingress throughput -10.64KiB/CPU-s -0.20 17.28% 5.22MiB/CPU-s 2.71MiB/CPU-s 34.15KiB/CPU-s 0.0 0.518831 5.21MiB/CPU-s 2.76MiB/CPU-s 34.78KiB/CPU-s 0.0 0.529684 True False
datadog_agent_remap_blackhole_acks ingress throughput -63.23KiB/CPU-s -0.21 98.50% 29.78MiB/CPU-s 1.65MiB/CPU-s 20.76KiB/CPU-s 0.0 0.055314 29.72MiB/CPU-s 1.24MiB/CPU-s 15.62KiB/CPU-s 0.0 0.041706 False False
syslog_splunk_hec_logs ingress throughput -51.38KiB/CPU-s -0.57 100.00% 8.87MiB/CPU-s 257.74KiB/CPU-s 3.17KiB/CPU-s 0.0 0.028371 8.82MiB/CPU-s 242.21KiB/CPU-s 2.98KiB/CPU-s 0.0 0.026813 False False
splunk_hec_route_s3 ingress throughput -72.04KiB/CPU-s -0.60 100.00% 11.66MiB/CPU-s 602.16KiB/CPU-s 7.41KiB/CPU-s 0.0 0.050441 11.59MiB/CPU-s 562.58KiB/CPU-s 6.92KiB/CPU-s 0.0 0.047411 False False
syslog_humio_logs ingress throughput -59.71KiB/CPU-s -0.64 100.00% 9.08MiB/CPU-s 282.27KiB/CPU-s 3.47KiB/CPU-s 0.0 0.030363 9.02MiB/CPU-s 261.63KiB/CPU-s 3.22KiB/CPU-s 0.0 0.028326 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -72.6KiB/CPU-s -0.76 100.00% 9.39MiB/CPU-s 209.19KiB/CPU-s 2.57KiB/CPU-s 0.0 0.021763 9.32MiB/CPU-s 280.68KiB/CPU-s 3.45KiB/CPU-s 0.0 0.029422 False False
syslog_log2metric_humio_metrics ingress throughput -60.43KiB/CPU-s -0.95 100.00% 6.19MiB/CPU-s 230.28KiB/CPU-s 2.83KiB/CPU-s 0.0 0.036356 6.13MiB/CPU-s 293.43KiB/CPU-s 3.61KiB/CPU-s 0.0 0.046773 False False
syslog_loki ingress throughput -189.54KiB/CPU-s -2.21 100.00% 8.38MiB/CPU-s 266.64KiB/CPU-s 3.28KiB/CPU-s 0.0 0.031073 8.19MiB/CPU-s 331.54KiB/CPU-s 4.08KiB/CPU-s 0.0 0.039509 False False

@github-actions
Copy link

github-actions bot commented Mar 31, 2023

Regression Detector Results

Run ID: 316807c2-c74c-49be-a2b6-001685663ff9
Baseline: 00c0316
Comparison: db2e52f
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole ingress throughput 576.62KiB/CPU-s 1.86 100.00% 30.3MiB/CPU-s 1.65MiB/CPU-s 20.82KiB/CPU-s 0.0 0.054556 30.86MiB/CPU-s 1.28MiB/CPU-s 16.1KiB/CPU-s 0.0 0.041385 False False
syslog_loki ingress throughput 146.06KiB/CPU-s 1.71 100.00% 8.37MiB/CPU-s 232.91KiB/CPU-s 2.86KiB/CPU-s 0.0 0.027186 8.51MiB/CPU-s 217.84KiB/CPU-s 2.68KiB/CPU-s 0.0 0.025001 False False
splunk_hec_route_s3 ingress throughput 107.03KiB/CPU-s 0.92 100.00% 11.33MiB/CPU-s 690.22KiB/CPU-s 8.49KiB/CPU-s 0.0 0.059474 11.44MiB/CPU-s 569.27KiB/CPU-s 7.01KiB/CPU-s 0.0 0.048604 False False
http_to_http_json ingress throughput 108.85KiB/CPU-s 0.79 100.00% 13.51MiB/CPU-s 351.87KiB/CPU-s 4.33KiB/CPU-s 0.0 0.025424 13.62MiB/CPU-s 226.98KiB/CPU-s 2.79KiB/CPU-s 0.0 0.016272 False False
otlp_grpc_to_blackhole ingress throughput 6.95KiB/CPU-s 0.67 100.00% 1.01MiB/CPU-s 52.0KiB/CPU-s 655.18B/CPU-s 0.0 0.050079 1.02MiB/CPU-s 41.23KiB/CPU-s 519.6B/CPU-s 0.0 0.03944 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 131.07KiB/CPU-s 0.39 100.00% 32.55MiB/CPU-s 1.16MiB/CPU-s 14.59KiB/CPU-s 0.0 0.035582 32.68MiB/CPU-s 937.57KiB/CPU-s 11.54KiB/CPU-s 0.0 0.028013 False False
http_to_http_acks ingress throughput 18.85KiB/CPU-s 0.35 30.25% 5.21MiB/CPU-s 2.72MiB/CPU-s 34.31KiB/CPU-s 0.0 0.523022 5.22MiB/CPU-s 2.72MiB/CPU-s 34.27KiB/CPU-s 0.0 0.52071 True False
datadog_agent_remap_datadog_logs ingress throughput 110.92KiB/CPU-s 0.33 100.00% 32.6MiB/CPU-s 876.9KiB/CPU-s 10.79KiB/CPU-s 0.0 0.026263 32.71MiB/CPU-s 1.26MiB/CPU-s 15.85KiB/CPU-s 0.0 0.038456 False False
http_text_to_http_json ingress throughput 57.51KiB/CPU-s 0.23 100.00% 24.1MiB/CPU-s 543.8KiB/CPU-s 6.69KiB/CPU-s 0.0 0.022032 24.16MiB/CPU-s 684.31KiB/CPU-s 8.42KiB/CPU-s 0.0 0.02766 False False
enterprise_http_to_http ingress throughput 7.52KiB/CPU-s 0.05 91.46% 13.61MiB/CPU-s 299.72KiB/CPU-s 3.69KiB/CPU-s 0.0 0.021496 13.62MiB/CPU-s 191.01KiB/CPU-s 2.35KiB/CPU-s 0.0 0.013692 False False
splunk_hec_indexer_ack_blackhole ingress throughput 2.04KiB/CPU-s 0.01 35.07% 13.62MiB/CPU-s 262.41KiB/CPU-s 3.23KiB/CPU-s 0.0 0.01882 13.62MiB/CPU-s 252.18KiB/CPU-s 3.1KiB/CPU-s 0.0 0.018084 False False
http_to_http_noack ingress throughput 1.75KiB/CPU-s 0.01 24.15% 13.61MiB/CPU-s 335.35KiB/CPU-s 4.13KiB/CPU-s 0.0 0.024058 13.61MiB/CPU-s 317.58KiB/CPU-s 3.91KiB/CPU-s 0.0 0.02278 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -366.04B/CPU-s -0.00 6.24% 13.62MiB/CPU-s 260.94KiB/CPU-s 3.21KiB/CPU-s 0.0 0.018715 13.62MiB/CPU-s 263.69KiB/CPU-s 3.24KiB/CPU-s 0.0 0.018912 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -66.27B/CPU-s -0.00 0.83% 13.61MiB/CPU-s 356.93KiB/CPU-s 4.39KiB/CPU-s 0.0 0.025602 13.61MiB/CPU-s 362.51KiB/CPU-s 4.46KiB/CPU-s 0.0 0.026002 False False
fluent_elasticsearch ingress throughput 284.85B/CPU-s 0.00 40.95% 45.41MiB/CPU-s 29.81KiB/CPU-s 371.45B/CPU-s 0.0 0.000641 45.41MiB/CPU-s 30.25KiB/CPU-s 377.2B/CPU-s 0.0 0.00065 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -10.23KiB/CPU-s -0.27 79.91% 3.68MiB/CPU-s 471.28KiB/CPU-s 5.8KiB/CPU-s 0.0 0.125081 3.67MiB/CPU-s 447.18KiB/CPU-s 5.5KiB/CPU-s 0.0 0.119007 True True
syslog_log2metric_splunk_hec_metrics ingress throughput -61.91KiB/CPU-s -0.66 100.00% 9.12MiB/CPU-s 364.76KiB/CPU-s 4.49KiB/CPU-s 0.0 0.039055 9.06MiB/CPU-s 419.53KiB/CPU-s 5.16KiB/CPU-s 0.0 0.045219 False False
syslog_splunk_hec_logs ingress throughput -60.72KiB/CPU-s -0.67 100.00% 8.89MiB/CPU-s 271.66KiB/CPU-s 3.34KiB/CPU-s 0.0 0.029836 8.83MiB/CPU-s 184.05KiB/CPU-s 2.26KiB/CPU-s 0.0 0.02035 False False
otlp_http_to_blackhole ingress throughput -11.27KiB/CPU-s -0.72 100.00% 1.53MiB/CPU-s 121.1KiB/CPU-s 1.49KiB/CPU-s 0.0 0.077445 1.52MiB/CPU-s 122.21KiB/CPU-s 1.5KiB/CPU-s 0.0 0.078724 False False
datadog_agent_remap_blackhole_acks ingress throughput -263.59KiB/CPU-s -0.83 100.00% 31.17MiB/CPU-s 921.12KiB/CPU-s 11.33KiB/CPU-s 0.0 0.028855 30.91MiB/CPU-s 1.26MiB/CPU-s 15.85KiB/CPU-s 0.0 0.040691 False False
syslog_log2metric_humio_metrics ingress throughput -52.53KiB/CPU-s -0.84 100.00% 6.13MiB/CPU-s 330.43KiB/CPU-s 4.07KiB/CPU-s 0.0 0.052628 6.08MiB/CPU-s 221.11KiB/CPU-s 2.72KiB/CPU-s 0.0 0.035513 False False
socket_to_socket_blackhole ingress throughput -128.29KiB/CPU-s -0.95 100.00% 13.2MiB/CPU-s 329.27KiB/CPU-s 4.05KiB/CPU-s 0.0 0.024364 13.07MiB/CPU-s 347.19KiB/CPU-s 4.27KiB/CPU-s 0.0 0.025936 False False
syslog_humio_logs ingress throughput -136.08KiB/CPU-s -1.47 100.00% 9.06MiB/CPU-s 228.87KiB/CPU-s 2.82KiB/CPU-s 0.0 0.024662 8.93MiB/CPU-s 224.04KiB/CPU-s 2.76KiB/CPU-s 0.0 0.024501 False False
file_to_blackhole egress throughput -106.01KiB/CPU-s -1.54 42.86% 6.71MiB/CPU-s 4.15MiB/CPU-s 128.06KiB/CPU-s 5.167724 0.618612 6.61MiB/CPU-s 4.65MiB/CPU-s 136.65KiB/CPU-s 0.0 0.70287 True True

@github-actions
Copy link

github-actions bot commented Mar 31, 2023

Regression Detector Results

Run ID: 91dee1ce-7b4a-4932-89a3-62f93302b19e
Baseline: 00c0316
Comparison: baf1b9e
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole_acks ingress throughput 811.01KiB/CPU-s 2.58 100.00% 30.75MiB/CPU-s 1.82MiB/CPU-s 22.99KiB/CPU-s 0.0 0.059331 31.54MiB/CPU-s 836.74KiB/CPU-s 10.3KiB/CPU-s 0.0 0.025908 False False
datadog_agent_remap_datadog_logs ingress throughput 813.39KiB/CPU-s 2.44 100.00% 32.5MiB/CPU-s 1.12MiB/CPU-s 14.06KiB/CPU-s 0.0 0.034325 33.3MiB/CPU-s 963.03KiB/CPU-s 11.85KiB/CPU-s 0.0 0.028241 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 745.46KiB/CPU-s 2.28 100.00% 31.96MiB/CPU-s 1.2MiB/CPU-s 15.12KiB/CPU-s 0.0 0.037553 32.69MiB/CPU-s 1.31MiB/CPU-s 16.45KiB/CPU-s 0.0 0.039943 False False
otlp_http_to_blackhole ingress throughput 24.28KiB/CPU-s 1.56 100.00% 1.52MiB/CPU-s 126.75KiB/CPU-s 1.56KiB/CPU-s 0.0 0.081208 1.55MiB/CPU-s 108.77KiB/CPU-s 1.34KiB/CPU-s 0.0 0.068621 False False
syslog_regex_logs2metric_ddmetrics ingress throughput 22.14KiB/CPU-s 0.62 99.85% 3.5MiB/CPU-s 413.08KiB/CPU-s 5.08KiB/CPU-s 0.0 0.115144 3.52MiB/CPU-s 385.92KiB/CPU-s 4.75KiB/CPU-s 0.0 0.106914 True True
splunk_hec_route_s3 ingress throughput 49.46KiB/CPU-s 0.41 100.00% 11.74MiB/CPU-s 528.61KiB/CPU-s 6.5KiB/CPU-s 0.0 0.043971 11.79MiB/CPU-s 511.97KiB/CPU-s 6.3KiB/CPU-s 0.0 0.042412 False False
http_text_to_http_json ingress throughput 36.26KiB/CPU-s 0.15 99.12% 23.79MiB/CPU-s 709.63KiB/CPU-s 8.73KiB/CPU-s 0.0 0.02913 23.82MiB/CPU-s 871.29KiB/CPU-s 10.72KiB/CPU-s 0.0 0.035713 False False
http_to_http_noack ingress throughput 5.64KiB/CPU-s 0.04 70.45% 13.61MiB/CPU-s 339.95KiB/CPU-s 4.18KiB/CPU-s 0.0 0.024389 13.62MiB/CPU-s 275.94KiB/CPU-s 3.4KiB/CPU-s 0.0 0.019788 False False
enterprise_http_to_http ingress throughput 3.77KiB/CPU-s 0.03 74.22% 13.62MiB/CPU-s 223.55KiB/CPU-s 2.75KiB/CPU-s 0.0 0.016027 13.62MiB/CPU-s 152.47KiB/CPU-s 1.88KiB/CPU-s 0.0 0.010928 False False
http_to_http_json ingress throughput -354.57B/CPU-s -0.00 6.32% 13.61MiB/CPU-s 252.23KiB/CPU-s 3.1KiB/CPU-s 0.0 0.018096 13.61MiB/CPU-s 249.72KiB/CPU-s 3.07KiB/CPU-s 0.0 0.017917 False False
fluent_elasticsearch ingress throughput 408.17B/CPU-s 0.00 55.76% 45.41MiB/CPU-s 29.81KiB/CPU-s 371.48B/CPU-s 0.0 0.000641 45.41MiB/CPU-s 30.5KiB/CPU-s 379.9B/CPU-s 0.0 0.000656 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -159.32B/CPU-s -0.00 1.90% 13.61MiB/CPU-s 373.37KiB/CPU-s 4.59KiB/CPU-s 0.0 0.02678 13.61MiB/CPU-s 378.4KiB/CPU-s 4.65KiB/CPU-s 0.0 0.027142 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -236.42B/CPU-s -0.00 4.08% 13.62MiB/CPU-s 259.52KiB/CPU-s 3.19KiB/CPU-s 0.0 0.018612 13.62MiB/CPU-s 259.89KiB/CPU-s 3.2KiB/CPU-s 0.0 0.018638 False False
splunk_hec_indexer_ack_blackhole ingress throughput -1.71KiB/CPU-s -0.01 30.21% 13.62MiB/CPU-s 247.57KiB/CPU-s 3.05KiB/CPU-s 0.0 0.017753 13.62MiB/CPU-s 259.37KiB/CPU-s 3.19KiB/CPU-s 0.0 0.018602 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -31.96KiB/CPU-s -0.33 100.00% 9.37MiB/CPU-s 256.09KiB/CPU-s 3.15KiB/CPU-s 0.0 0.026684 9.34MiB/CPU-s 247.86KiB/CPU-s 3.05KiB/CPU-s 0.0 0.025913 False False
otlp_grpc_to_blackhole ingress throughput -9.75KiB/CPU-s -0.93 100.00% 1.02MiB/CPU-s 47.02KiB/CPU-s 592.55B/CPU-s 0.0 0.045081 1.01MiB/CPU-s 56.49KiB/CPU-s 711.72B/CPU-s 0.0 0.054663 False False
socket_to_socket_blackhole ingress throughput -128.62KiB/CPU-s -0.94 100.00% 13.32MiB/CPU-s 361.95KiB/CPU-s 4.45KiB/CPU-s 0.0 0.026538 13.19MiB/CPU-s 356.23KiB/CPU-s 4.38KiB/CPU-s 0.0 0.026367 False False
datadog_agent_remap_blackhole ingress throughput -294.08KiB/CPU-s -0.94 100.00% 30.62MiB/CPU-s 1.25MiB/CPU-s 15.79KiB/CPU-s 0.0 0.040945 30.33MiB/CPU-s 1.4MiB/CPU-s 17.66KiB/CPU-s 0.0 0.046194 False False
http_to_http_acks ingress throughput -65.37KiB/CPU-s -1.22 82.43% 5.24MiB/CPU-s 2.68MiB/CPU-s 33.81KiB/CPU-s 0.0 0.511673 5.18MiB/CPU-s 2.73MiB/CPU-s 34.45KiB/CPU-s 0.0 0.527934 True False
file_to_blackhole egress throughput -87.42KiB/CPU-s -1.23 36.58% 6.93MiB/CPU-s 4.09MiB/CPU-s 131.85KiB/CPU-s 0.0 0.590377 6.85MiB/CPU-s 4.43MiB/CPU-s 127.89KiB/CPU-s 0.397141 0.647073 True True
syslog_splunk_hec_logs ingress throughput -155.92KiB/CPU-s -1.72 100.00% 8.87MiB/CPU-s 267.43KiB/CPU-s 3.29KiB/CPU-s 0.0 0.029427 8.72MiB/CPU-s 269.67KiB/CPU-s 3.32KiB/CPU-s 0.0 0.030191 False False
syslog_humio_logs ingress throughput -219.52KiB/CPU-s -2.36 100.00% 9.07MiB/CPU-s 237.17KiB/CPU-s 2.92KiB/CPU-s 0.0 0.025532 8.86MiB/CPU-s 338.82KiB/CPU-s 4.17KiB/CPU-s 0.0 0.037357 False False
syslog_loki ingress throughput -224.37KiB/CPU-s -2.62 100.00% 8.37MiB/CPU-s 274.78KiB/CPU-s 3.38KiB/CPU-s 0.0 0.032073 8.15MiB/CPU-s 255.03KiB/CPU-s 3.14KiB/CPU-s 0.0 0.030569 False False
syslog_log2metric_humio_metrics ingress throughput -208.41KiB/CPU-s -3.21 100.00% 6.33MiB/CPU-s 180.3KiB/CPU-s 2.22KiB/CPU-s 0.0 0.0278 6.13MiB/CPU-s 289.02KiB/CPU-s 3.56KiB/CPU-s 0.0 0.046043 False False

@neuronull neuronull requested a review from jszwedko March 31, 2023 22:40
@neuronull neuronull marked this pull request as ready for review March 31, 2023 22:42
@neuronull neuronull merged commit 11588da into master Apr 3, 2023
@neuronull neuronull deleted the neuronull/custom_builds_int_tests_fix branch April 3, 2023 17:38
@jszwedko jszwedko mentioned this pull request Apr 12, 2023
Merged
@neuronull neuronull mentioned this pull request Aug 17, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jszwedko jszwedko Awaiting requested review from jszwedko

Assignees

@neuronull neuronull

Labels

domain: ci Anything related to Vector's CI environment domain: releasing Anything related to releasing Vector

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@neuronull @jszwedko