[O11y][Kubernetes] Rally benchmark `kubernetes.event` by ali786XI · Pull Request #9161 · elastic/integrations

ali786XI · 2024-02-15T05:43:36Z

Proposed commit message

This PR adds benchmarking templates to the event data stream of Kubernetes

Sample Response

sample_event.json

{
"kubernetes": {
  "event": {
    "reason": "NodeHasSufficientMemory",
    "metadata": {
      "uid": "host-48",
      "generate_name": "treerazor",
      "name": "host-48",
      "namespace": "demo-48",
      "annotations": {
        "kubectl_kubernetes_io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"involvedObject\":{\"kind\":\"Pod\"},\"kind\":\"Event\",\"message\":\"Node minikube status is now: NodeHasSufficientMemory\",\"metadata\":{\"annotations\":{},\"name\":\"host-48\",\"namespace\":\"demo-48\"},\"type\":\"Normal\"}\n"
      },
      "self_link": "/api/v1/namespaces/events/host-48",
      "resource_version": "481307",
      "timestamp": {
        "created": "2024-02-22T08:32:43.416+05:30"
      }
    },
    "involved_object": {
      "uid": "host-48",
      "kind": "Pod",
      "name": "boldlizard",
      "api_version": "v1",
      "resource_version": "481307"
    },
    "count": 4693,
    "source": {
      "component": "demo-component-48",
      "host": "host-48"
    },
    "message": "Node minikube status is now: NodeHasSufficientMemory",
    "type": "Normal",
    "timestamp": {
      "first_occurrence": "2024-02-22T08:32:43.416+05:30",
      "last_occurrence": "2024-02-22T12:12:43.416+05:30"
    }
  }
},
"agent": {
  "name": "kubernetes-scale-123456",
  "id": "de42127b-4db8-4471-824e-a7b14f478663",
  "ephemeral_id": "22ed892c-43bd-408a-9121-65e2f5b6a56e",
  "type": "metricbeat",
  "version": "8.8.0"
},
"benchmark_metadata": {
  "info": {
    "run_id": "60d2bff6-ec81-4bd0-b83a-52ac8bd1e7e3",
    "benchmark": "event-benchmark"
  }
},
"elastic_agent": {
  "id": "de42127b-4db8-4471-824e-a7b14f478663",
  "version": "8.8.0",
  "snapshot": true
},
"orchestrator": {
  "cluster": {
    "name": "kubernetes-scale",
    "url": "https://22.110.111.190"
  }
},
"@timestamp": "2024-02-22T12:32:43.416+05:30",
"ecs": {
  "version": "8.5.0"
},
"data_stream": {
  "namespace": "ep",
  "type": "metrics",
  "dataset": "kubernetes.event"
},
"service": {
  "type": "kubernetes"
},
"host": {
  "hostname": "kubernetes-scale-123456",
  "os": {
    "kernel": "5.10.47-linuxkit",
    "codename": "Core",
    "name": "CentOS Linux",
    "family": "redhat",
    "type": "linux",
    "version": "7 (Core)",
    "platform": "centos"
  },
  "containerized": true,
  "name": "kubernetes-scale-123456",
  "id": "85e35c2b5e1b39ba72393a6baf6ee7cd",
  "architecture": "x86_64"
},
"metricset": {
  "name": "event"
},
"event": {
  "duration": 2288442,
  "agent_id_status": "auth_metadata_missing",
  "ingested": "2024-02-22T07:02:45Z",
  "module": "kubernetes",
  "dataset": "kubernetes.event"
}
}

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

elastic-package benchmark rally --benchmark event-benchmark -v
elastic-package benchmark stream --benchmark event-benchmark -v

Related issues

Relates [Meta] Add more benchmark schema templates to integrations #8636

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                               │
├────────────────────────┬───────────────────────────────────────────────────────────┤
│ benchmark              │                                           event-benchmark │
│ description            │        Benchmark of 20000 kubernetes.event event ingested │
│ run ID                 │                      46dce708-a1fd-4e36-9da7-683a5c820cde │
│ package                │                                                kubernetes │
│ start ts (s)           │                                                1707975134 │
│ end ts (s)             │                                                1707975168 │
│ duration               │                                                       34s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-1298637713 │
╰────────────────────────┴───────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────╮
│ parameters                                                          │
├─────────────────────────────────┬───────────────────────────────────┤
│ package version                 │                            1.56.0 │
│ data_stream.name                │                             event │
│ corpora.generator.total_events  │                             20000 │
│ corpora.generator.template.path │ ./event-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                   │
│ corpora.generator.template.type │                            gotext │
│ corpora.generator.config.path   │      ./event-benchmark/config.yml │
│ corpora.generator.config.raw    │                             map[] │
│ corpora.generator.fields.path   │      ./event-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                             map[] │
╰─────────────────────────────────┴───────────────────────────────────╯
╭───────────╮
│ cluster i │
│ nfo       │
├───────┬───┤
│ name  │   │
│ nodes │ 0 │
╰───────┴───╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-metrics-kube │
│ rnetes.event-ep-2024.02.15-000001 (fo │
│ r all fields)                         │
├──────────────────────────────┬────────┤
│ total                        │ 2.0 MB │
│ inverted_index.total         │ 186 kB │
│ inverted_index.stored_fields │ 1.1 MB │
│ inverted_index.doc_values    │ 339 kB │
│ inverted_index.points        │ 385 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭───────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-kubernetes.event-1.56.0 stats in node iMbidc_lRGGO7pfdllYYfw     │
├────────────────────────────────────────────┬──────────────────────────────────────┤
│ Totals                                     │ Count: 20000 | Failed: 0 | Time: 5ms │
│ pipeline (metrics-kubernetes.event@custom) │ Count: 20000 | Failed: 0 | Time: 2ms │
╰────────────────────────────────────────────┴──────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │     1.5145333333333333 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │    0.04186666666666667 min │
│ Max cumulative indexing time across primary shards             │    0.26483333333333337 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │                  0.491 min │
│ Cumulative merge count of primary shards                       │                        387 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │   0.005908333333333333 min │
│ Max cumulative merge time across primary shards                │    0.06848333333333333 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │    0.36795000000000005 min │
│ Cumulative refresh count of primary shards                     │                      12444 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │   0.015250000000000001 min │
│ Max cumulative refresh time across primary shards              │   0.047150000000000004 min │
│ Cumulative flush time of primary shards                        │      4.910283333333334 min │
│ Cumulative flush count of primary shards                       │                      12226 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │    0.11680833333333333 min │
│ Max cumulative flush time across primary shards                │     0.3876833333333333 min │
│ Total Young Gen GC time                                        │                    0.022 s │
│ Total Young Gen GC count                                       │                          3 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │    0.042072431184351444 GB │
│ Translog size                                                  │    0.040557156316936016 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        305 │
│ Total Ingest Pipeline count                                    │                      20026 │
│ Total Ingest Pipeline time                                     │                    0.977 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │             4403.14 docs/s │
│ Mean Throughput                                                │             4696.60 docs/s │
│ Median Throughput                                              │             4740.67 docs/s │
│ Max Throughput                                                 │             4901.94 docs/s │
│ 50th percentile latency                                        │       90.52182408049703 ms │
│ 90th percentile latency                                        │      106.79234878625722 ms │
│ 100th percentile latency                                       │      119.29205409251153 ms │
│ 50th percentile service time                                   │       90.52182408049703 ms │
│ 90th percentile service time                                   │      106.79234878625722 ms │
│ 100th percentile service time                                  │      119.29205409251153 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

elasticmachine · 2024-02-15T06:21:13Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

gizas

Only major thing that I spotted is the data_stream is missing

    "data_stream": {
      "namespace": "default",
      "type": "metrics",
      "dataset": "kubernetes.event"
    }

You can also add the kubectl_kubernetes_io/last-applied-configuration annotation

Adding here the sample event I created for test
event.json.txt

gizas · 2024-02-22T08:45:45Z

packages/kubernetes/_dev/benchmark/rally/event-benchmark/template.ndjson

+        "module": "kubernetes"
+    },
+    "data_stream": {
+      "namespace": "ep",


Suggested change

"namespace": "ep",

"namespace": "default",

Was done on purpose?

Oh forgot to mention this. Actually keeping the namepsace as default results in this error while streaming the metrics
field [data_stream.namespace] only accepts values that are equal to the value defined in the mappings [ep], but got [default]"}}}}. So changed it to ep

… into kubernetes_benchmark_event

elasticmachine · 2024-02-26T11:32:57Z

💚 Build Succeeded

Buildkite Build
Commit: db96f67

History

💚 Build #9189 succeeded 3219231
💚 Build #9135 succeeded b2d8c8f
💚 Build #9084 succeeded 12b204f
💚 Build #9057 succeeded c6f8327
💚 Build #8984 succeeded 3bb6e22

cc @aliabbas-elastic

elastic-sonarqube · 2024-02-26T11:33:00Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

rally benchmark kubernetes.event

3bb6e22

ali786XI added enhancement New feature or request Integration:kubernetes Kubernetes labels Feb 15, 2024

ali786XI self-assigned this Feb 15, 2024

ali786XI mentioned this pull request Feb 15, 2024

[Meta] Add more benchmark schema templates to integrations #8636

Closed

29 tasks

ali786XI added 2 commits February 19, 2024 09:47

remove some fields

c6f8327

refactor fields

12b204f

ali786XI marked this pull request as ready for review February 20, 2024 06:46

ali786XI requested a review from a team as a code owner February 20, 2024 06:46

ali786XI requested review from MichaelKatsoulis, gizas and gsantoro February 20, 2024 06:46

gizas requested changes Feb 21, 2024

View reviewed changes

ali786XI added 2 commits February 21, 2024 16:10

add data stream fields

b2d8c8f

add last-applied-configuration

3219231

ali786XI requested a review from gizas February 22, 2024 07:08

gizas reviewed Feb 22, 2024

View reviewed changes

gizas approved these changes Feb 22, 2024

View reviewed changes

Merge branch 'main' of https://github.com/aliabbas-elastic/integrations…

db96f67

… into kubernetes_benchmark_event

ali786XI merged commit 4ba7ee4 into elastic:main Feb 26, 2024

gizas pushed a commit that referenced this pull request Mar 13, 2024

[O11y][Kubernetes] Rally benchmark kubernetes.event (#9161)

64bbbaa

qcorporation pushed a commit that referenced this pull request Feb 3, 2025

[O11y][Kubernetes] Rally benchmark kubernetes.event (#9161)

4c5756c

qcorporation pushed a commit that referenced this pull request Feb 4, 2025

[O11y][Kubernetes] Rally benchmark kubernetes.event (#9161)

56c25a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[O11y][Kubernetes] Rally benchmark `kubernetes.event`#9161

[O11y][Kubernetes] Rally benchmark `kubernetes.event`#9161
ali786XI merged 6 commits intoelastic:mainfrom
ali786XI:kubernetes_benchmark_event

ali786XI commented Feb 15, 2024 •

edited

Loading

Uh oh!

elasticmachine commented Feb 15, 2024

Uh oh!

gizas left a comment

Uh oh!

gizas Feb 22, 2024

Uh oh!

ali786XI Feb 22, 2024

Uh oh!

elasticmachine commented Feb 26, 2024

Uh oh!

elastic-sonarqube bot commented Feb 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ali786XI commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Sample Response

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Feb 15, 2024

🚀 Benchmarks report

Uh oh!

gizas left a comment

Choose a reason for hiding this comment

Uh oh!

gizas Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

ali786XI Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 26, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Feb 26, 2024

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ali786XI commented Feb 15, 2024 •

edited

Loading