[O11y][Nginx] Rally benchmark `nginx.stubstatus` by ali786XI · Pull Request #8800 · elastic/integrations

ali786XI · 2023-12-27T13:43:02Z

Proposed commit message

This PR adds benchmarking templates to the stubstatus data stream of Nginx

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

elastic-package benchmark rally --benchmark stubstatus-benchmark -v

Related issues

Relates [Rally] Add benchmarking templates to data streams for set of packages #8719

Screenshots

--- Benchmark results for package: nginx - START ---
╭───────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                              │
├────────────────────────┬──────────────────────────────────────────────────────────┤
│ benchmark              │                                     stubstatus-benchmark │
│ description            │         Benchmark 20000 nginx.stubstatus events ingested │
│ run ID                 │                     17a28197-6467-4f2c-8f64-c27df499d26a │
│ package                │                                                    nginx │
│ start ts (s)           │                                               1703240060 │
│ end ts (s)             │                                               1703240093 │
│ duration               │                                                      33s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-968077677 │
╰────────────────────────┴──────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────╮
│ parameters                                                               │
├─────────────────────────────────┬────────────────────────────────────────┤
│ package version                 │                                  1.9.0 │
│ data_stream.name                │                             stubstatus │
│ corpora.generator.total_events  │                                  20000 │
│ corpora.generator.template.path │ ./stubstatus-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                        │
│ corpora.generator.template.type │                                 gotext │
│ corpora.generator.config.path   │      ./stubstatus-benchmark/config.yml │
│ corpora.generator.config.raw    │                                  map[] │
│ corpora.generator.fields.path   │      ./stubstatus-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                                  map[] │
╰─────────────────────────────────┴────────────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭──────────────────────────────────────────────────────────╮
│ data stream stats                                        │
├────────────────────────────┬─────────────────────────────┤
│ data stream                │ metrics-nginx.stubstatus-ep │
│ approx total docs ingested │                       20000 │
│ backing indices            │                           2 │
│ store size bytes           │                     4255645 │
│ maximum ts (ms)            │               1703259855887 │
╰────────────────────────────┴─────────────────────────────╯
╭────────────────────────────────────╮
│ disk usage for index .ds-metrics-n │
│ ginx.stubstatus-ep-2023.12.22-0000 │
│ 01 (for all fields)                │
├──────────────────────────────┬─────┤
│ total                        │ 0 B │
│ inverted_index.total         │ 0 B │
│ inverted_index.stored_fields │ 0 B │
│ inverted_index.doc_values    │ 0 B │
│ inverted_index.points        │ 0 B │
│ inverted_index.norms         │ 0 B │
│ inverted_index.term_vectors  │ 0 B │
│ inverted_index.knn_vectors   │ 0 B │
╰──────────────────────────────┴─────╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-metrics-ngin │
│ x.stubstatus-ep-2023.12.22-000002 (fo │
│ r all fields)                         │
├──────────────────────────────┬────────┤
│ total                        │ 4.1 MB │
│ inverted_index.total         │ 918 kB │
│ inverted_index.stored_fields │ 1.8 MB │
│ inverted_index.doc_values    │ 982 kB │
│ inverted_index.points        │ 408 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭───────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-nginx.stubstatus-1.9.0 stats in node WTEal3z2TIGOWeDz4OkCXA      │
├────────────────────────────────────────────┬──────────────────────────────────────┤
│ Totals                                     │ Count: 20000 | Failed: 0 | Time: 5ms │
│ pipeline (metrics-nginx.stubstatus@custom) │ Count: 20000 | Failed: 0 | Time: 2ms │
╰────────────────────────────────────────────┴──────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                │
├────────────────────────────────────────────────────────────────┬───────────────────────────┤
│ Cumulative indexing time of primary shards                     │   0.23173333333333332 min │
│ Min cumulative indexing time across primary shards             │                     0 min │
│ Median cumulative indexing time across primary shards          │                0.0009 min │
│ Max cumulative indexing time across primary shards             │   0.16523333333333332 min │
│ Cumulative indexing throttle time of primary shards            │                     0 min │
│ Min cumulative indexing throttle time across primary shards    │                     0 min │
│ Median cumulative indexing throttle time across primary shards │                   0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                     0 min │
│ Cumulative merge time of primary shards                        │  0.012466666666666666 min │
│ Cumulative merge count of primary shards                       │                         9 │
│ Min cumulative merge time across primary shards                │                     0 min │
│ Median cumulative merge time across primary shards             │                   0.0 min │
│ Max cumulative merge time across primary shards                │                0.0021 min │
│ Cumulative merge throttle time of primary shards               │                     0 min │
│ Min cumulative merge throttle time across primary shards       │                     0 min │
│ Median cumulative merge throttle time across primary shards    │                   0.0 min │
│ Max cumulative merge throttle time across primary shards       │                     0 min │
│ Cumulative refresh time of primary shards                      │   0.08023333333333334 min │
│ Cumulative refresh count of primary shards                     │                       719 │
│ Min cumulative refresh time across primary shards              │                     0 min │
│ Median cumulative refresh time across primary shards           │ 0.0008083333333333333 min │
│ Max cumulative refresh time across primary shards              │  0.031716666666666664 min │
│ Cumulative flush time of primary shards                        │    0.2059666666666667 min │
│ Cumulative flush count of primary shards                       │                       445 │
│ Min cumulative flush time across primary shards                │                     0 min │
│ Median cumulative flush time across primary shards             │  0.006666666666666667 min │
│ Max cumulative flush time across primary shards                │  0.017833333333333333 min │
│ Total Young Gen GC time                                        │                   0.025 s │
│ Total Young Gen GC count                                       │                         2 │
│ Total Old Gen GC time                                          │                       0 s │
│ Total Old Gen GC count                                         │                         0 │
│ Store size                                                     │   0.015282240696251392 GB │
│ Translog size                                                  │     0.0226362906396389 GB │
│ Heap used for segments                                         │                      0 MB │
│ Heap used for doc values                                       │                      0 MB │
│ Heap used for terms                                            │                      0 MB │
│ Heap used for norms                                            │                      0 MB │
│ Heap used for points                                           │                      0 MB │
│ Heap used for stored fields                                    │                      0 MB │
│ Segment count                                                  │                       425 │
│ Total Ingest Pipeline count                                    │                     20022 │
│ Total Ingest Pipeline time                                     │                   0.682 s │
│ Total Ingest Pipeline failed                                   │                         0 │
│ Min Throughput                                                 │            4915.02 docs/s │
│ Mean Throughput                                                │            5779.72 docs/s │
│ Median Throughput                                              │            6043.72 docs/s │
│ Max Throughput                                                 │            6380.42 docs/s │
│ 50th percentile latency                                        │      66.57252553850412 ms │
│ 90th percentile latency                                        │      76.21758494060487 ms │
│ 100th percentile latency                                       │     152.68584992736578 ms │
│ 50th percentile service time                                   │      66.57252553850412 ms │
│ 90th percentile service time                                   │      76.21758494060487 ms │
│ 100th percentile service time                                  │     152.68584992736578 ms │
│ error rate                                                     │                    0.00 % │
╰────────────────────────────────────────────────────────────────┴───────────────────────────╯

--- Benchmark results for package: nginx - END   ---
Done

packages/nginx/_dev/benchmark/rally/stubstatus-benchmark/template.ndjson

elasticmachine · 2023-12-28T13:01:08Z

🚀 Benchmarks report

Package `nginx` 👍(1) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`error`	23255.81	12345.68	-10910.13 (-46.91%)	💔

To see the full report comment with /test benchmark fullreport

tommyers-elastic · 2023-12-29T10:03:31Z

packages/nginx/_dev/benchmark/rally/stubstatus-benchmark/config.yml

+  - name: agent.id
+    value: "ef5e274d-4b53-45e6-943a-a5bcf1a6f523"
+  - name: host.name
+    cardinality: 100


we suffer from the same 'linked' cardinality issue here, which would cause pretty much all the subsequent fields to be static for a given host.

i really don't like the way this works, but for now we have to work around it. we are discussing the best way how in this thread.

we suffer from the same 'linked' cardinality issue here, which would cause pretty much all the subsequent fields to be static for a given host.

i really don't like the way this works, but for now we have to work around it.

@tommyers-elastic do you mind creating an issue about what are the current limit you see when needed to link each other different fields?

cardinality is the practical way we have at the moment, it was not introduced with the goal to solve the problem about linked fields (and it does not even fit every scenario, like with the introduction of counter)

elastic/elastic-integration-corpus-generator-tool#128

tommyers-elastic

LGTM

niraj-elastic

LGTM!

ruflin · 2024-01-17T08:08:43Z

If I'm running this with elastic-package benchmark stream --benchmark stubstatus-benchmark -v I get the following error:

2024/01/17 09:07:44 DEBUG bulk request of -15m0s backfill events on metrics-nginx.stubstatus-ep...
Error: error running package stream benchmarks: could not complete benchmark run: error performing bulk request: 900 failed

@aliabbas-elastic Does this work on your end?

ali786XI · 2024-01-17T08:24:26Z

If I'm running this with elastic-package benchmark stream --benchmark stubstatus-benchmark -v I get the following error:
2024/01/17 09:07:44 DEBUG bulk request of -15m0s backfill events on metrics-nginx.stubstatus-ep...
Error: error running package stream benchmarks: could not complete benchmark run: error performing bulk request: 900 failed
@aliabbas-elastic Does this work on your end?

@ruflin I confirm this is happening on my end as well. I think this is coming due to TSDB enablement on the stubstatus data stream and removing the setting works well. But this should not be the case and I am checking how to resolve this error.

ruflin · 2024-01-17T10:09:29Z

Did you find some more details around the error you get?

@aspacca Would be great to support these "debugging issues" use case better with something like elastic/elastic-package#1610

rally benchmark nginx.stubstatus

7599c86

ali786XI self-assigned this Dec 27, 2023

ali786XI added enhancement New feature or request Integration:nginx Nginx labels Dec 27, 2023

ali786XI commented Dec 27, 2023

View reviewed changes

packages/nginx/_dev/benchmark/rally/stubstatus-benchmark/template.ndjson Show resolved Hide resolved

remove repeated fields

f0bad0a

ali786XI mentioned this pull request Dec 27, 2023

[Rally] Add benchmarking templates to data streams for set of packages #8719

Closed

ali786XI marked this pull request as ready for review December 27, 2023 15:49

ali786XI requested a review from a team as a code owner December 27, 2023 15:49

ali786XI requested a review from tommyers-elastic December 27, 2023 15:49

harnish-crest-data self-requested a review December 28, 2023 06:29

remove generate for event.ingested and host.ip

bd092d3

tommyers-elastic reviewed Dec 29, 2023

View reviewed changes

ali786XI mentioned this pull request Jan 4, 2024

[Meta] Add more benchmark schema templates to integrations #8636

Closed

29 tasks

relace host name with service address field

84c54b9

ali786XI requested a review from tommyers-elastic January 10, 2024 06:23

remove cardinality and fuzziness

ef4d542

ali786XI removed the request for review from harnish-crest-data January 16, 2024 13:03

tommyers-elastic approved these changes Jan 16, 2024

View reviewed changes

niraj-elastic approved these changes Jan 17, 2024

View reviewed changes

ali786XI merged commit 455719b into elastic:main Jan 17, 2024

ruflin mentioned this pull request Jan 17, 2024

Validate command for benchmark templates elastic/elastic-package#1610

Open

ali786XI mentioned this pull request Jan 17, 2024

[Nginx] Fix stubstatus benchmark #8914

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[O11y][Nginx] Rally benchmark `nginx.stubstatus`#8800

[O11y][Nginx] Rally benchmark `nginx.stubstatus`#8800
ali786XI merged 5 commits intoelastic:mainfrom
ali786XI:nginx_benchmark_stubstatus

ali786XI commented Dec 27, 2023

Uh oh!

Uh oh!

elasticmachine commented Dec 28, 2023 •

edited

Loading

Uh oh!

tommyers-elastic Dec 29, 2023

Uh oh!

francescayeye Jan 18, 2024

Uh oh!

tommyers-elastic Jan 18, 2024

Uh oh!

tommyers-elastic left a comment

Uh oh!

niraj-elastic left a comment

Uh oh!

ruflin commented Jan 17, 2024

Uh oh!

ali786XI commented Jan 17, 2024

Uh oh!

ruflin commented Jan 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ali786XI commented Dec 27, 2023

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

elasticmachine commented Dec 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package nginx 👍(1) 💚(0) 💔(1)

Uh oh!

tommyers-elastic Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

francescayeye Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

niraj-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

ruflin commented Jan 17, 2024

Uh oh!

ali786XI commented Jan 17, 2024

Uh oh!

ruflin commented Jan 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

elasticmachine commented Dec 28, 2023 •

edited

Loading

Package `nginx` 👍(1) 💚(0) 💔(1)