[O11y][Kubernetes] Rally benchmark `kubernetes.state_container` by ali786XI · Pull Request #9106 · elastic/integrations

ali786XI · 2024-02-09T10:14:45Z

Proposed commit message

This PR adds benchmarking templates to the state_container data stream of Kubernetes

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

elastic-package benchmark rally --benchmark state_container-benchmark -v
elastic-package benchmark stream --benchmark state_container-benchmark -v

Related issues

Relates [Meta] Add more benchmark schema templates to integrations #8636

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭─────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                                │
├────────────────────────┬────────────────────────────────────────────────────────────┤
│ benchmark              │                                   statecontainer-benchmark │
│ description            │ Benchmark 20000 kubernetes.state_container events ingested │
│ run ID                 │                       0b22e23e-d5f3-4055-91c9-17441ec98e37 │
│ package                │                                                 kubernetes │
│ start ts (s)           │                                                 1707473600 │
│ end ts (s)             │                                                 1707473651 │
│ duration               │                                                        51s │
│ generated corpora file │  /root/.elastic-package/tmp/rally_corpus/corpus-1816164413 │
╰────────────────────────┴────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────╮
│ parameters                                                                   │
├─────────────────────────────────┬────────────────────────────────────────────┤
│ package version                 │                                     1.56.0 │
│ data_stream.name                │                            state_container │
│ corpora.generator.total_events  │                                      20000 │
│ corpora.generator.template.path │ ./statecontainer-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                            │
│ corpora.generator.template.type │                                     gotext │
│ corpora.generator.config.path   │      ./statecontainer-benchmark/config.yml │
│ corpora.generator.config.raw    │                                      map[] │
│ corpora.generator.fields.path   │      ./statecontainer-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                                      map[] │
╰─────────────────────────────────┴────────────────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭───────────────────────────────────────╮
│ disk usage for index metrics-kubernet │
│ es.state_container-ep (for all fields │
│ )                                     │
├──────────────────────────────┬────────┤
│ total                        │  12 MB │
│ inverted_index.total         │ 1.9 MB │
│ inverted_index.stored_fields │ 7.4 MB │
│ inverted_index.doc_values    │ 2.5 MB │
│ inverted_index.points        │ 345 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-kubernetes.state_container-1.56.0 stats in node OQo_iVeVRSeYfWHrhZJ5_g      │
├──────────────────────────────────────────────────────┬───────────────────────────────────────┤
│ Totals                                               │ Count: 20000 | Failed: 0 | Time: 27ms │
│ pipeline (global@custom)                             │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics@custom)                            │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes@custom)                 │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes.state_container@custom) │  Count: 20000 | Failed: 0 | Time: 2ms │
╰──────────────────────────────────────────────────────┴───────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │     0.2747833333333333 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │  0.0005750000000000001 min │
│ Max cumulative indexing time across primary shards             │    0.19011666666666666 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │                      0 min │
│ Cumulative merge count of primary shards                       │                          0 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                    0.0 min │
│ Max cumulative merge time across primary shards                │                      0 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │    0.05616666666666667 min │
│ Cumulative refresh count of primary shards                     │                        543 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │                 0.0005 min │
│ Max cumulative refresh time across primary shards              │   0.016483333333333332 min │
│ Cumulative flush time of primary shards                        │    0.16481666666666667 min │
│ Cumulative flush count of primary shards                       │                        362 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │               0.005125 min │
│ Max cumulative flush time across primary shards                │                0.02075 min │
│ Total Young Gen GC time                                        │                    0.067 s │
│ Total Young Gen GC count                                       │                          5 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │    0.022678245790302753 GB │
│ Translog size                                                  │   0.0001719137653708458 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        395 │
│ Total Ingest Pipeline count                                    │                      20068 │
│ Total Ingest Pipeline time                                     │                    1.629 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │            21020.27 docs/s │
│ Mean Throughput                                                │            21020.27 docs/s │
│ Median Throughput                                              │            21020.27 docs/s │
│ Max Throughput                                                 │            21020.27 docs/s │
│ 50th percentile latency                                        │       884.0546769788489 ms │
│ 100th percentile latency                                       │       894.0900478046387 ms │
│ 50th percentile service time                                   │       884.0546769788489 ms │
│ 100th percentile service time                                  │       894.0900478046387 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

elasticmachine · 2024-02-09T10:54:18Z

🚀 Benchmarks report

Package `aws` 👍(11) 💚(3) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`apigateway_logs`	13888.89	11627.91	-2260.98 (-16.28%)	💔
`elb_logs`	5464.48	3021.15	-2443.33 (-44.71%)	💔
`emr_logs`	25641.03	18867.92	-6773.11 (-26.42%)	💔

Package `mysql` 👍(0) 💚(0) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`error`	14925.37	7092.2	-7833.17 (-52.48%)	💔
`slowlog`	20833.33	13333.33	-7500 (-36%)	💔

To see the full report comment with /test benchmark fullreport

gizas · 2024-02-12T10:01:30Z

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson

+        "name": "state_container"
+    },
+    "event": {
+        "duration": {{ $event_duration }},


ingested fields is missing
eg. event.ingested: "2024-02-08T12:18:41Z"

Please refer here for the same doubt I had. It's taking the time actually when the event is ingested.

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson

gizas · 2024-02-12T10:02:33Z

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson

+                "phase": "{{ $status_phase }}",
+                "ready": {{ $status_ready }},
+                "restarts": {{ $restarts }},
+                "reason": "{{ $reason }}"


Suggested change

"reason": "{{ $reason }}"

"last_terminated_reason": "{{ $reason }}"

Took reference from here. Found this field you are mentioning as kubernetes.container.status.last_terminated_reason

I am testing with a real cluster and just spotted that as well.

In beats: https://github.com/elastic/beats/blob/main/metricbeat/module/kubernetes/state_container/state_container.go#L69

Will have a second look in the background to explain the diff

When container is in terminated or waiting state ,the status.reason is populated
When container is in running state, then the status kubernetes.container.status.last_terminated_reason is populated

See info here and here

So it is probably another if case for you, that when phase is running then populate this kubernetes.container.status.last_terminated_reason

@gizas Thanks for this. Added the same.

gizas · 2024-02-12T10:04:46Z

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson

+            },
+            "id": "container-{{ $rangeofid }}",
+            "status": {
+                "phase": "{{ $status_phase }}",


The only values that can probably cause us problems in the future are these one for the status. Because the way it is implemented now a 'running' phase can have ready: false , because those values are randomly assigned.

Can you consider an if case like https://github.com/elastic/elastic-integration-corpus-generator-tool/blob/main/assets/templates/aws.billing/schema-b/gotext.tpl#L64 ?

Sure. Updated the same. Please have a look.

gizas · 2024-02-12T10:08:00Z

@aliabbas-elastic I have tested them and work fine. Thanks for this. Added some comments.

Also because the dataset is named state_container, I would also advise that all folders/paths should be renamed to state_container-benchmark and change to state_container-benchmark.yml

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/config.yml

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/fields.yml

ali786XI · 2024-02-12T13:27:39Z

Also because the dataset is named state_container, I would also advise that all folders/paths should be renamed to state_container-benchmark and change to state_container-benchmark.yml

@gizas Yes initially kept the name as you mentioned but facing these below error in linting

Lint the package
2024/02/12 18:54:13 Warning: references found in dashboard kibana/dashboard/kubernetes-f4dc26db-1b53-4ea2-a78b-1bfab8ea267c.json: kubernetes-ee55101a-9f62-44da-b64c-ffa1eb5abad8 (search) (SVR00004)
Error: checking package failed: linting package failed: found 1 validation error:
   1. item [state_container-benchmark] is not allowed in folder [/aliabbas-elastic/integrations/packages/kubernetes/_dev/benchmark/rally]

Saw couple of data streams as well with the present naming convention. Keeping the naming as per you suggested is definitely better. Is this a bug in the check command?
cc:- @aspacca

gizas · 2024-02-12T14:18:55Z

Also because the dataset is named state_container, I would also advise that all folders/paths should be renamed to state_container-benchmark and change to state_container-benchmark.yml

@gizas Yes initially kept the name as you mentioned but facing these below error in linting

You are right. I could not find in elastic-package where this is done. in my tests:

#Changing the folder to state_container-benchmark

elastic-package lint
Error: linting package failed: found 1 validation error:
   1. item [state_container-benchmark] is not allowed in folder [/Users/andreasgkizas/elastic/integrations/packages/kubernetes/_dev/benchmark/rally]

Changing the file only and NOT the folder to state_container-benchmark.yml

❯ elastic-package lint
2024/02/12 16:13:18  INFO New version is available - v0.97.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.97.0
Lint the package
2024/02/12 16:13:19 Warning: references found in dashboard kibana/dashboard/kubernetes-f4dc26db-1b53-4ea2-a78b-1bfab8ea267c.json: kubernetes-ee55101a-9f62-44da-b64c-ffa1eb5abad8 (search) (SVR00004)
Done

Then the command becomes:
❯ elastic-package benchmark rally --benchmark state_container-benchmark

Dont you think is better? Or creates more confusion?

ali786XI · 2024-02-12T14:27:29Z

Dont you think is better? Or creates more confusion?

Sounds better than the previous as there will be no ambiguity in running a benchmark without looking at the benchmark structure.

francescayeye · 2024-02-13T01:09:38Z

@aliabbas-elastic . @gizas

the limit on folder naming is by package-spec:
https://github.com/elastic/package-spec/blob/main/spec/integration/_dev/benchmark/spec.yml#L26-L62

folder: pattern: '^[a-z0-9]+-benchmark$'

no limit on the scenario name: pattern: '^.+\.yml$'

it's just a matter of changing the specs, please proceed as you see it fitting :)

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson

elasticmachine · 2024-02-14T16:15:07Z

💚 Build Succeeded

Buildkite Build
Commit: 74a8e64

History

💚 Build #8871 succeeded c821c27
💚 Build #8866 succeeded 8c02a6f
💚 Build #8865 succeeded 988e6c4
💚 Build #8809 succeeded a9ad8d0

cc @aliabbas-elastic

elastic-sonarqube · 2024-02-14T16:15:08Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

rally benchmark kubernetes.state_container

07c98bb

ali786XI added enhancement New feature or request Integration:kubernetes Kubernetes labels Feb 9, 2024

ali786XI self-assigned this Feb 9, 2024

add container name

a9ad8d0

ali786XI changed the title ~~rally benchmark kubernetes.state_container~~ [O11y][Kubernetes] Rally benchmark kubernetes.state_container Feb 9, 2024

ali786XI mentioned this pull request Feb 9, 2024

[Meta] Add more benchmark schema templates to integrations #8636

Closed

29 tasks

ali786XI marked this pull request as ready for review February 12, 2024 05:52

ali786XI requested a review from a team as a code owner February 12, 2024 05:52

ali786XI requested review from gizas, gsantoro, harnish-crest-data and tommyers-elastic February 12, 2024 05:52

gizas reviewed Feb 12, 2024

View reviewed changes

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson Show resolved Hide resolved

gizas reviewed Feb 12, 2024

View reviewed changes

tommyers-elastic reviewed Feb 12, 2024

View reviewed changes

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/config.yml Outdated Show resolved Hide resolved

tommyers-elastic reviewed Feb 12, 2024

View reviewed changes

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/fields.yml Show resolved Hide resolved

address review comments

988e6c4

ali786XI requested review from gizas and tommyers-elastic February 12, 2024 13:29

add last_terminated_reason

8c02a6f

gizas approved these changes Feb 12, 2024

View reviewed changes

rename yml file

27d02ea

ali786XI force-pushed the kubernetes_benchmark_statecontainer branch from a4ff42c to 27d02ea Compare February 12, 2024 15:02

rename yml files for aws package

4da0f68

ali786XI requested review from a team as code owners February 12, 2024 15:04

rename yml files for mysql package

c821c27

harnish-crest-data approved these changes Feb 14, 2024

View reviewed changes

tommyers-elastic approved these changes Feb 14, 2024

View reviewed changes

gizas reviewed Feb 14, 2024

View reviewed changes

packages/kubernetes/_dev/benchmark/rally/statecontainer-benchmark/template.ndjson Outdated Show resolved Hide resolved

remove host.ip and host.mac

74a8e64

ali786XI merged commit 2829b1c into elastic:main Feb 15, 2024

andrewkroh added Integration:mysql MySQL Integration:aws AWS labels Jul 22, 2024

	"reason": "{{ $reason }}"
	"last_terminated_reason": "{{ $reason }}"

Conversation

ali786XI commented Feb 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Feb 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(11) 💚(3) 💔(3)

Package mysql 👍(0) 💚(0) 💔(2)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gizas commented Feb 12, 2024

Uh oh!

Uh oh!

Uh oh!

ali786XI commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gizas commented Feb 12, 2024

Uh oh!

ali786XI commented Feb 12, 2024

Uh oh!

francescayeye commented Feb 13, 2024

Uh oh!

Uh oh!

elasticmachine commented Feb 14, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Feb 14, 2024

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ali786XI commented Feb 9, 2024 •

edited

Loading

elasticmachine commented Feb 9, 2024 •

edited

Loading

Package `aws` 👍(11) 💚(3) 💔(3)

Package `mysql` 👍(0) 💚(0) 💔(2)

ali786XI commented Feb 12, 2024 •

edited

Loading