Skip to content

[O11y][System] Rally benchmark system.diskio#9295

Merged
shmsr merged 2 commits intoelastic:mainfrom
ali786XI:system_benchmark_diskio
Apr 3, 2024
Merged

[O11y][System] Rally benchmark system.diskio#9295
shmsr merged 2 commits intoelastic:mainfrom
ali786XI:system_benchmark_diskio

Conversation

@ali786XI
Copy link
Copy Markdown
Contributor

@ali786XI ali786XI commented Mar 7, 2024

Proposed commit message

  • This PR adds benchmarking templates to the diskio data stream of System

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

  • elastic-package benchmark rally --benchmark diskio-benchmark -v
  • elastic-package benchmark stream --benchmark diskio-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: system - START ---
╭────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                               │
├────────────────────────┬───────────────────────────────────────────────────────────┤
│ benchmark              │                                          diskio-benchmark │
│ description            │             Benchmark 20000 system.diskio events ingested │
│ run ID                 │                      383615d3-6174-4b1a-a59c-c5873c46af7d │
│ package                │                                                    system │
│ start ts (s)           │                                                1709795003 │
│ end ts (s)             │                                                1709795048 │
│ duration               │                                                       45s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-2041252408 │
╰────────────────────────┴───────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────╮
│ parameters                                                           │
├─────────────────────────────────┬────────────────────────────────────┤
│ package version                 │                             1.54.0 │
│ data_stream.name                │                             diskio │
│ corpora.generator.total_events  │                              20000 │
│ corpora.generator.template.path │ ./diskio-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                    │
│ corpora.generator.template.type │                             gotext │
│ corpora.generator.config.path   │      ./diskio-benchmark/config.yml │
│ corpora.generator.config.raw    │                              map[] │
│ corpora.generator.fields.path   │      ./diskio-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                              map[] │
╰─────────────────────────────────┴────────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-metrics-syst │
│ em.diskio-ep-2024.03.07-000001 (for a │
│ ll fields)                            │
├──────────────────────────────┬────────┤
│ total                        │ 3.9 MB │
│ inverted_index.total         │ 139 kB │
│ inverted_index.stored_fields │ 2.7 MB │
│ inverted_index.doc_values    │ 776 kB │
│ inverted_index.points        │ 274 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭─────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-system.diskio-1.54.0 stats in node G4sD7D1lRwyrYYoI76F0wQ      │
├─────────────────────────────────────────┬───────────────────────────────────────┤
│ Totals                                  │ Count: 20000 | Failed: 0 | Time: 28ms │
│ pipeline (global@custom)                │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics@custom)               │  Count: 20000 | Failed: 0 | Time: 3ms │
│ pipeline (metrics-system@custom)        │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-system.diskio@custom) │  Count: 20000 | Failed: 0 | Time: 2ms │
╰─────────────────────────────────────────┴───────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │                 0.3905 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │ 0.00019166666666666667 min │
│ Max cumulative indexing time across primary shards             │    0.25406666666666666 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │   0.005783333333333333 min │
│ Cumulative merge count of primary shards                       │                         10 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                    0.0 min │
│ Max cumulative merge time across primary shards                │  0.0012666666666666666 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │    0.07886666666666667 min │
│ Cumulative refresh count of primary shards                     │                        716 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │  8.333333333333333e-05 min │
│ Max cumulative refresh time across primary shards              │    0.01951666666666667 min │
│ Cumulative flush time of primary shards                        │                 0.1695 min │
│ Cumulative flush count of primary shards                       │                        450 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │                  5e-05 min │
│ Max cumulative flush time across primary shards                │   0.025983333333333334 min │
│ Total Young Gen GC time                                        │                    0.043 s │
│ Total Young Gen GC count                                       │                          3 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │    0.008164550177752972 GB │
│ Translog size                                                  │   0.0006286948919296265 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        168 │
│ Total Ingest Pipeline count                                    │                      20004 │
│ Total Ingest Pipeline time                                     │                    0.763 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ 100th percentile latency                                       │      14.955237973481417 ms │
│ 100th percentile service time                                  │      14.955237973481417 ms │
│ error rate                                                     │                   100.00 % │
│ Min Throughput                                                 │            41660.27 docs/s │
│ Mean Throughput                                                │            41660.27 docs/s │
│ Median Throughput                                              │            41660.27 docs/s │
│ Max Throughput                                                 │            41660.27 docs/s │
│ 50th percentile latency                                        │      435.90910569764674 ms │
│ 100th percentile latency                                       │       576.4065086841583 ms │
│ 50th percentile service time                                   │      435.90910569764674 ms │
│ 100th percentile service time                                  │       576.4065086841583 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: system - END   ---
Done

@ali786XI ali786XI added enhancement New feature or request Integration:system System labels Mar 7, 2024
@ali786XI ali786XI self-assigned this Mar 7, 2024
@ali786XI ali786XI mentioned this pull request Mar 7, 2024
Closed
29 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Mar 7, 2024

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@ali786XI ali786XI marked this pull request as ready for review March 7, 2024 08:22
@ali786XI ali786XI requested a review from a team as a code owner March 7, 2024 08:22
harnish-crest-data
harnish-crest-data approved these changes Mar 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@harnish-crest-data harnish-crest-data left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Mar 13, 2024

💚 Build Succeeded

History

cc @aliabbas-elastic

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Mar 13, 2024

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@shmsr shmsr merged commit 9e8fa77 into elastic:main Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shmsr shmsr shmsr approved these changes

@tommyers-elastic tommyers-elastic Awaiting requested review from tommyers-elastic

+1 more reviewer

@harnish-crest-data harnish-crest-data harnish-crest-data approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ali786XI ali786XI

Labels

enhancement New feature or request Integration:system System

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ali786XI @elasticmachine @shmsr @harnish-crest-data