Skip to content

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations#12535

Merged
gizas merged 4968 commits intomainfrom
k8senablement9.0
Feb 5, 2025
Merged

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations#12535
gizas merged 4968 commits intomainfrom
k8senablement9.0

Conversation

@gizas
Copy link
Copy Markdown
Contributor

@gizas gizas commented Jan 30, 2025

  • Enhancement

Dont merge until I run once the tests as described here
We will need to put only 9.0.0 in constrains, trigger the buidkite pipeline and then revert again

Tested with command https://github.com/elastic/integrations/pull/12535#issuecomment-2627285200 : /test stack 9.0.0-SNAPSHOT

Proposed commit message

  • WHAT: Enabling support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations
  • WHY: Is needed in order to enable above integrations in version 9.0.0

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  1. Clone Pr
  2. elastic-package build with v0.109.1
  3. elastic-package stack up -d -v --version=9.0.0-SNAPSHOT to install a local ES
  4. Install an agent + Fleet with prementioned integrations enabled

Related issues

Logs

Below validation errors fixed for istio and containerd
Containerd

Error: building package failed: invalid content found in built zip package: found 7 validation errors:
   1. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/blkio/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   2. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/cpu/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   3. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/memory/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   4. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field (root): Additional property license is not allowed
   5. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field (root): Additional property release is not allowed
   6. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field owner: type is required
   7. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field conditions: Additional property kibana.version is not allowed


Istio:

Error: building package failed: invalid content found in built zip package: found 6 validation errors:
   1. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/data_stream/access_logs/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.47.remove.field: rename "message" to "event.original" processor requires remove "message" processor (JSE00001)
   2. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/data_stream/access_logs/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.47.remove.if: rename "message" to "event.original" processor requires remove "message" processor with if: 'ctx.event?.original != null' (JSE00001)
   3. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field (root): Additional property release is not allowed
   4. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field (root): Additional property license is not allowed
   5. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field owner: type is required
   6. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field conditions: Additional property kibana.version is not allowed

Screenshots

K8s

![k8s1](https://github.com/user-attachments/assets/1366461a-1fd6-43

k8s2
6d-af4e-cfebfe13e0ff)

K8s Otel

k8s0tel

Containerd

containerd

Istio

istiod

Ingress-Controller

ingress

Overall

Screenshot 2025-01-30 at 4 39 30 PM

SimonKoetting and others added 30 commits December 17, 2024 08:22
Change property connection_string to be a secret like in the other integrations.
* Fix broken links

* Update changelog and manifest
Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…0.108.0 (#12131)

Bumps [github.com/elastic/elastic-package](https://github.com/elastic/elastic-package) from 0.107.2 to 0.108.0.
- [Release notes](https://github.com/elastic/elastic-package/releases)
- [Changelog](https://github.com/elastic/elastic-package/blob/main/.goreleaser.yml)
- [Commits](elastic/elastic-package@v0.107.2...v0.108.0)

---
updated-dependencies:
- dependency-name: github.com/elastic/elastic-package
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
Changes added:
- Add a limit parameter, that can be used to control the size of responses from TAXII servers (see https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107517)
- To avoid fetching duplicate indicators every interval, now the response header X-Taxii-Date-Added-Last is stored in the cursor and used to populate the added_after parameter every iteration (see https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107519)
* Update link

* Update changelog and manifest
#11920)

This is enabled per data stream to allow tuning of behaviour.
…nt" tag to documents with event.kind set to "pipeline_error" (#12108)

This manually replays the changes in #12046.
…at (#12110)

* Include pipeline test examples to accommodate the new Cloudtrail format
Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- Correct the `dg_time` format (may have AM/PM).
- Correct the `dg_processed_time` format (to `epoch_millis`).
- Process additional timestamp fields: `dg_local_timestamp`, `pi_fal`,
  `pi_fcl`, `pi_fml`.
- Add fallback/alternate formats for the string-valued time fields.
- For data size fields, add parsing logic.
…ual_policies mode (#12115)

* [Search Connectors] Use policy_templates_behavior option with individual_policies mode

* Update PR in changelog

* Re-trigger the build pipeline
Change property connection_string to be a secret like in the other integrations.
Added to the manifests of both data streams.
Already used by the input config for the `access` data stream.
Usage added in the input config for the `waf` data stream.
* Fix typos in SQL Input package
* add and update dashboards

* update changelog entry

* update dashboards

* update kibana version

* update changelog entry

* add dashboards screen shots

* fix changelog

* Update packages/vsphere/changelog.yml

Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>

* update remaining dashboards with id changes

* update kibana version to 8.16.2

* Hosts Overview

* Cluster and Network Status Insights

* Comprehensive Storage Overview

* Resource Pools Overview

* VMs overview

* update dashboards and its screen shots

---------

Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>
Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The API documentation for `vulnerability` mentions the `lastSeen` parameter format should be `#:#` where `#` is in number of days.
Although the existing format `@-@` where `@` is epoch seconds is still working, this discrepancy is addressed to avoid any future issues.

Other changes:
- Updated request's `header.User-Agent` versions in input files.
* add observability category
- Add initial implementation of the syslog router integration which
will identify and route syslog-based events to security integrations.
gizas added 6 commits February 5, 2025 12:31
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
@pierrehilbert
Copy link
Copy Markdown
Contributor

@gizas Looks like you have a lot of conflicts to resolve here, could you please have a look?

gizas added 5 commits February 5, 2025 14:42
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
@gizas
Copy link
Copy Markdown
Contributor Author

gizas commented Feb 5, 2025

/test stack 9.0.0-SNAPSHOT

@gizas
Copy link
Copy Markdown
Contributor Author

gizas commented Feb 5, 2025

thanks @pierrehilbert , now looks better

@elasticmachine
Copy link
Copy Markdown

⏳ Build in-progress, with failures

Failed CI Steps

History

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>
@gizas
Copy link
Copy Markdown
Contributor Author

gizas commented Feb 5, 2025

/test stack 9.0.0-SNAPSHOT

@elastic-sonarqube
Copy link
Copy Markdown

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

@gizas gizas merged commit 043460a into main Feb 5, 2025
@gizas gizas deleted the k8senablement9.0 branch February 5, 2025 15:32
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package containerd - 0.5.0 containing this change is available at https://epr.elastic.co/package/containerd/0.5.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package istio - 0.7.0 containing this change is available at https://epr.elastic.co/package/istio/0.7.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package kubernetes - 1.80.0 containing this change is available at https://epr.elastic.co/package/kubernetes/1.80.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package kubernetes_otel - 1.1.0 containing this change is available at https://epr.elastic.co/package/kubernetes_otel/1.1.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package nginx_ingress_controller - 1.11.0 containing this change is available at https://epr.elastic.co/package/nginx_ingress_controller/1.11.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:containerd Containerd Integration:istio Istio Integration:kubernetes_otel Kubernetes OpenTelemetry Assets Integration:kubernetes Kubernetes Integration:nginx_ingress_controller Nginx Ingress Controller Logs Team:Cloudnative-Monitoring Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]

Projects

None yet

Development

Successfully merging this pull request may close these issues.