Skip to content

[O11y][Kubernetes] Rally benchmark kubernetes.node#9100

Merged
ali786XI merged 4 commits intoelastic:mainfrom
ali786XI:kubernetes_benchmark_node
Feb 26, 2024
Merged

[O11y][Kubernetes] Rally benchmark kubernetes.node#9100
ali786XI merged 4 commits intoelastic:mainfrom
ali786XI:kubernetes_benchmark_node

Conversation

@ali786XI
Copy link
Copy Markdown
Contributor

@ali786XI ali786XI commented Feb 8, 2024 •
edited
Loading

Proposed commit message

  • This PR adds benchmarking templates to the node data stream of Kubernetes

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

  • elastic-package benchmark rally --benchmark node-benchmark -v
  • elastic-package benchmark stream --benchmark node-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭──────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                             │
├────────────────────────┬─────────────────────────────────────────────────────────┤
│ benchmark              │                                          node-benchmark │
│ description            │         Benchmark 20000 kubernetes.node events ingested │
│ run ID                 │                    65325741-6e4f-4493-8008-2f6bd9b0e546 │
│ package                │                                              kubernetes │
│ start ts (s)           │                                                       0 │
│ end ts (s)             │                                              1707473229 │
│ duration               │                                             474298h7m9s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-33646997 │
╰────────────────────────┴─────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────╮
│ parameters                                                         │
├─────────────────────────────────┬──────────────────────────────────┤
│ package version                 │                           1.56.0 │
│ data_stream.name                │                             node │
│ corpora.generator.total_events  │                            20000 │
│ corpora.generator.template.path │ ./node-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                  │
│ corpora.generator.template.type │                           gotext │
│ corpora.generator.config.path   │      ./node-benchmark/config.yml │
│ corpora.generator.config.raw    │                            map[] │
│ corpora.generator.fields.path   │      ./node-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                            map[] │
╰─────────────────────────────────┴──────────────────────────────────╯
╭───────────╮
│ cluster i │
│ nfo       │
├───────┬───┤
│ name  │   │
│ nodes │ 1 │
╰───────┴───╯
╭───────────────────────────────────────╮
│ disk usage for index metrics-kubernet │
│ es.node-ep (for all fields)           │
├──────────────────────────────┬────────┤
│ total                        │ 9.8 MB │
│ inverted_index.total         │ 182 kB │
│ inverted_index.stored_fields │ 7.1 MB │
│ inverted_index.doc_values    │ 2.0 MB │
│ inverted_index.points        │ 474 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                │
├────────────────────────────────────────────────────────────────┬───────────────────────────┤
│ Cumulative indexing time of primary shards                     │   0.20473333333333335 min │
│ Min cumulative indexing time across primary shards             │                     0 min │
│ Median cumulative indexing time across primary shards          │ 0.0005333333333333334 min │
│ Max cumulative indexing time across primary shards             │   0.12678333333333333 min │
│ Cumulative indexing throttle time of primary shards            │                     0 min │
│ Min cumulative indexing throttle time across primary shards    │                     0 min │
│ Median cumulative indexing throttle time across primary shards │                   0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                     0 min │
│ Cumulative merge time of primary shards                        │                     0 min │
│ Cumulative merge count of primary shards                       │                         0 │
│ Min cumulative merge time across primary shards                │                     0 min │
│ Median cumulative merge time across primary shards             │                   0.0 min │
│ Max cumulative merge time across primary shards                │                     0 min │
│ Cumulative merge throttle time of primary shards               │                     0 min │
│ Min cumulative merge throttle time across primary shards       │                     0 min │
│ Median cumulative merge throttle time across primary shards    │                   0.0 min │
│ Max cumulative merge throttle time across primary shards       │                     0 min │
│ Cumulative refresh time of primary shards                      │   0.05068333333333333 min │
│ Cumulative refresh count of primary shards                     │                       415 │
│ Min cumulative refresh time across primary shards              │                     0 min │
│ Median cumulative refresh time across primary shards           │ 0.0004333333333333333 min │
│ Max cumulative refresh time across primary shards              │  0.014566666666666667 min │
│ Cumulative flush time of primary shards                        │   0.10021666666666666 min │
│ Cumulative flush count of primary shards                       │                       254 │
│ Min cumulative flush time across primary shards                │                     0 min │
│ Median cumulative flush time across primary shards             │                0.0031 min │
│ Max cumulative flush time across primary shards                │               0.00785 min │
│ Total Young Gen GC time                                        │                   0.088 s │
│ Total Young Gen GC count                                       │                         6 │
│ Total Old Gen GC time                                          │                       0 s │
│ Total Old Gen GC count                                         │                         0 │
│ Store size                                                     │   0.029821311123669147 GB │
│ Translog size                                                  │    0.04626855626702309 GB │
│ Heap used for segments                                         │                      0 MB │
│ Heap used for doc values                                       │                      0 MB │
│ Heap used for terms                                            │                      0 MB │
│ Heap used for norms                                            │                      0 MB │
│ Heap used for points                                           │                      0 MB │
│ Heap used for stored fields                                    │                      0 MB │
│ Segment count                                                  │                       301 │
│ Total Ingest Pipeline count                                    │                     20037 │
│ Total Ingest Pipeline time                                     │                   2.987 s │
│ Total Ingest Pipeline failed                                   │                         0 │
│ Min Throughput                                                 │            1996.24 docs/s │
│ Mean Throughput                                                │            1996.24 docs/s │
│ Median Throughput                                              │            1996.24 docs/s │
│ Max Throughput                                                 │            1996.24 docs/s │
│ 50th percentile latency                                        │     1151.2397719779983 ms │
│ 100th percentile latency                                       │     1215.1859279256314 ms │
│ 50th percentile service time                                   │     1151.2397719779983 ms │
│ 100th percentile service time                                  │     1215.1859279256314 ms │
│ error rate                                                     │                    0.00 % │
╰────────────────────────────────────────────────────────────────┴───────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

@ali786XI ali786XI added enhancement New feature or request Integration:kubernetes Kubernetes labels Feb 8, 2024
@ali786XI ali786XI self-assigned this Feb 8, 2024
@ali786XI ali786XI requested a review from a team February 8, 2024 06:04
@ali786XI ali786XI mentioned this pull request Feb 8, 2024
Closed
29 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 8, 2024 •
edited
Loading

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_logs 125000 90909.09 -34090.91 (-27.27%) 💔

To see the full report comment with /test benchmark fullreport

@ali786XI ali786XI requested review from a team and removed request for a team February 8, 2024 06:36
tommyers-elastic
tommyers-elastic approved these changes Feb 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see any issues here, but it would be useful to see diffs of the generated events for these PRs compared with say the sample event in the package, to quickly verify any missing or significantly different fields.

@ali786XI ali786XI marked this pull request as ready for review February 14, 2024 13:13
@ali786XI ali786XI requested a review from a team as a code owner February 14, 2024 13:13
gizas
gizas requested changes Feb 14, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@gizas gizas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cc a node document from the cluster:
node.json

Related to comment here for removal : #9100 (comment)

@ali786XI
Copy link
Copy Markdown
Contributor Author

ali786XI commented Feb 14, 2024

I cc a node document from the cluster:
node.json

Thanks for sharing the sample response @gizas. I have updated it accordingly. Let me know if the set of fields removed are correct.

@ali786XI ali786XI requested a review from gizas February 14, 2024 15:29
@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Feb 14, 2024

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Feb 14, 2024

💚 Build Succeeded

History

cc @aliabbas-elastic

@ali786XI ali786XI merged commit e191f83 into elastic:main Feb 26, 2024
gizas pushed a commit that referenced this pull request Mar 13, 2024
@ali786XI @gizas
[O11y][Kubernetes] Rally benchmark kubernetes.node (#9100)
99880e1
qcorporation pushed a commit that referenced this pull request Feb 3, 2025
@ali786XI @gizas
[O11y][Kubernetes] Rally benchmark kubernetes.node (#9100)
3ecfbea
qcorporation pushed a commit that referenced this pull request Feb 4, 2025
@ali786XI @gizas
[O11y][Kubernetes] Rally benchmark kubernetes.node (#9100)
9b5410c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gizas gizas gizas approved these changes

@tommyers-elastic tommyers-elastic tommyers-elastic approved these changes

@tetianakravchenko tetianakravchenko Awaiting requested review from tetianakravchenko tetianakravchenko was automatically assigned from elastic/obs-cloudnative-monitoring

+1 more reviewer

@milan-elastic milan-elastic milan-elastic approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ali786XI ali786XI

Labels

enhancement New feature or request Integration:kubernetes Kubernetes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ali786XI @elasticmachine @gizas @tommyers-elastic @milan-elastic