Skip to content

[O11y][Kubernetes] Rally benchmark kubernetes.state_pod#9136

Merged
ali786XI merged 3 commits intoelastic:mainfrom
ali786XI:kubernetes_benchmark_statepod
Feb 15, 2024
Merged

[O11y][Kubernetes] Rally benchmark kubernetes.state_pod#9136
ali786XI merged 3 commits intoelastic:mainfrom
ali786XI:kubernetes_benchmark_statepod

Conversation

@ali786XI
Copy link
Copy Markdown
Contributor

@ali786XI ali786XI commented Feb 13, 2024

Proposed commit message

  • This PR adds benchmarking templates to the state_pod data stream of Kubernetes

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

  • elastic-package benchmark rally --benchmark state_pod-benchmark -v
  • elastic-package benchmark stream --benchmark state_pod-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                               │
├────────────────────────┬───────────────────────────────────────────────────────────┤
│ benchmark              │                                       state_pod-benchmark │
│ description            │      Benchmark 20000 kubernetes.state_pod events ingested │
│ run ID                 │                      4c270392-77a3-4c11-875a-4ee173e29829 │
│ package                │                                                kubernetes │
│ start ts (s)           │                                                1707812774 │
│ end ts (s)             │                                                1707812820 │
│ duration               │                                                       46s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-1190356990 │
╰────────────────────────┴───────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────╮
│ parameters                                                             │
├─────────────────────────────────┬──────────────────────────────────────┤
│ package version                 │                               1.56.0 │
│ data_stream.name                │                            state_pod │
│ corpora.generator.total_events  │                                20000 │
│ corpora.generator.template.path │ ./statepod-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                      │
│ corpora.generator.template.type │                               gotext │
│ corpora.generator.config.path   │      ./statepod-benchmark/config.yml │
│ corpora.generator.config.raw    │                                map[] │
│ corpora.generator.fields.path   │      ./statepod-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                                map[] │
╰─────────────────────────────────┴──────────────────────────────────────╯
╭───────────╮
│ cluster i │
│ nfo       │
├───────┬───┤
│ name  │   │
│ nodes │ 0 │
╰───────┴───╯
╭───────────────────────────────────────╮
│ disk usage for index metrics-kubernet │
│ es.state_pod-ep (for all fields)      │
├──────────────────────────────┬────────┤
│ total                        │ 5.9 MB │
│ inverted_index.total         │ 928 kB │
│ inverted_index.stored_fields │ 3.5 MB │
│ inverted_index.doc_values    │ 1.1 MB │
│ inverted_index.points        │ 438 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-kubernetes.state_pod-1.56.0 stats in node OQo_iVeVRSeYfWHrhZJ5_g      │
├────────────────────────────────────────────────┬───────────────────────────────────────┤
│ Totals                                         │ Count: 20000 | Failed: 0 | Time: 29ms │
│ pipeline (global@custom)                       │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics@custom)                      │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes@custom)           │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes.state_pod@custom) │  Count: 20000 | Failed: 0 | Time: 4ms │
╰────────────────────────────────────────────────┴───────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │                8.46765 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │    0.08625833333333334 min │
│ Max cumulative indexing time across primary shards             │     2.1194333333333333 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │      4.946816666666667 min │
│ Cumulative merge count of primary shards                       │                       2783 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                    0.0 min │
│ Max cumulative merge time across primary shards                │     0.6093833333333334 min │
│ Cumulative merge throttle time of primary shards               │                 0.0598 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                 0.0598 min │
│ Cumulative refresh time of primary shards                      │               117.6942 min │
│ Cumulative refresh count of primary shards                     │                      98008 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │    0.04891666666666667 min │
│ Max cumulative refresh time across primary shards              │               37.39995 min │
│ Cumulative flush time of primary shards                        │     226.82090000000002 min │
│ Cumulative flush count of primary shards                       │                      86269 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │   0.004808333333333333 min │
│ Max cumulative flush time across primary shards                │      47.68103333333333 min │
│ Total Young Gen GC time                                        │                    0.048 s │
│ Total Young Gen GC count                                       │                          4 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │     0.24722508061677217 GB │
│ Translog size                                                  │   0.0002751322463154793 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        642 │
│ Total Ingest Pipeline count                                    │                      20037 │
│ Total Ingest Pipeline time                                     │                    1.444 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │            24548.56 docs/s │
│ Mean Throughput                                                │            24548.56 docs/s │
│ Median Throughput                                              │            24548.56 docs/s │
│ Max Throughput                                                 │            24548.56 docs/s │
│ 50th percentile latency                                        │       743.7380615156144 ms │
│ 100th percentile latency                                       │       1083.487429190427 ms │
│ 50th percentile service time                                   │       743.7380615156144 ms │
│ 100th percentile service time                                  │       1083.487429190427 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

@ali786XI ali786XI added enhancement New feature or request Integration:kubernetes Kubernetes labels Feb 13, 2024
@ali786XI ali786XI self-assigned this Feb 13, 2024
@ali786XI ali786XI mentioned this pull request Feb 13, 2024
Closed
29 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 13, 2024 •
edited
Loading

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_logs 125000 55555.56 -69444.44 (-55.56%) 💔

To see the full report comment with /test benchmark fullreport

@ali786XI ali786XI marked this pull request as ready for review February 13, 2024 09:39
@ali786XI ali786XI requested a review from a team as a code owner February 13, 2024 09:39
gizas
gizas reviewed Feb 14, 2024
View reviewed changes
packages/kubernetes/_dev/benchmark/rally/statepod-benchmark/template.ndjson
"name": "demo-deployment-{{ $rangeofid }}"
},
"namespace_labels": {
"kubernetes_io/metadata_name": "demo-{{ $rangeofid }}"
Copy link
Copy Markdown
Contributor

@gizas gizas Feb 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont see a definition of rangeofid in config.go like here https://github.com/elastic/integrations/blob/main/packages/kubernetes/_dev/benchmark/rally/pod-benchmark/config.yml#L36

Is not it better in order to have specific range?

Copy link
Copy Markdown
Contributor Author

@ali786XI ali786XI Feb 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas Thanks for pointing out. I missed it.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 14, 2024

💚 Build Succeeded

History

cc @aliabbas-elastic

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Feb 14, 2024

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@ali786XI ali786XI merged commit e300934 into elastic:main Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gizas gizas gizas approved these changes

@constanca-m constanca-m constanca-m approved these changes

+1 more reviewer

@harnish-crest harnish-crest harnish-crest approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ali786XI ali786XI

Labels

enhancement New feature or request Integration:kubernetes Kubernetes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ali786XI @elasticmachine @gizas @harnish-crest @constanca-m