Add a readme for k8s autodiscover provider#28213

Merged

MichaelKatsoulis merged 9 commits intoelastic:masterfrom

MichaelKatsoulis:k8s-autodiscover-tech-doc

Jan 11, 2022

Contributor

MichaelKatsoulis commented Oct 1, 2021

What does this PR do?

This PR creates a markdown file explaining the high and low level functionality of Kubernetes autodiscover provider.
It consists of three major segments.

Templates Based Autodiscover
Hints Based Autodiscover
Autodiscover provider in Elastic Agent

Why is it important?

The intention is to have a technical document which will help everyone who wants to understand how Kubernetes autodiscover works.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.


          Add a readme for k8s autodiscover provider

96b3aa2

MichaelKatsoulis added docs Team:Integrations labels

Contributor

elasticmachine commented Oct 1, 2021

Pinging @elastic/integrations (Team:Integrations)

botelastic bot added needs_team and removed needs_team labels

Contributor

mergify bot commented Oct 1, 2021

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

mergify bot added the backport-skip label


          Fix duplicated line

addd263

MichaelKatsoulis removed the Team:Integrations label

botelastic bot added the needs_team label

MichaelKatsoulis requested a review from ChrsMark

October 1, 2021 11:40

Contributor

elasticmachine commented Oct 1, 2021 •

edited by jenkins-beats-ci bot

Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-10T14:50:08.043+0000
Duration: 20 min 4 sec
Commit: 4e89dbb

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

ChrsMark reviewed

View reviewed changes

Member

ChrsMark left a comment

Very nice that we finally have tech documentation for this codebase!
I left some suggestions about wording. Also moving Agent's part to the respective codebase directory is something that seems more suitable.

Also some comments:

leader_election is described as part of libbeat's provider but we have also a dedicated provider for Agent. Maybe it is worth documenting it to since you are on it.
In addition to this, there is also the kubernetes_secrets provider for Agent and kubernetes_secrtes backend for Beats, which is something we could document too as part of this effort.

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated

+              ### Template based autodiscover
+              Example:
+              As en example we will use gain redis module.

Member

ChrsMark Oct 3, 2021

Suggested change

      
            As en example we will use gain redis module.
          
            As en example we will use again redis module.

libbeat/autodiscover/README.md Outdated

+              ### Template based autodiscover
+              Example:
+              As en example we will use gain redis module.

Member

ChrsMark Oct 3, 2021

Suggested change

      
            As en example we will use gain redis module.
          
            As en example we will use gain the `redis` module.

libbeat/autodiscover/README.md Outdated

+              As en example we will use gain redis module.
+              In agent.yml(configmap) an extra input block needs to be added.
+              ```
+                    # Add extra input blocks here, based on conditions

Member

ChrsMark Oct 3, 2021

indentation?

Contributor Author

MichaelKatsoulis Oct 25, 2021

no need

libbeat/autodiscover/README.md




		## Autodiscover provider in Elastic Agent

Member

ChrsMark Oct 3, 2021

I would suggest moving this part under elastic-agent codebase so as to be close to the provider's code. Then you can cross-link it from libbeat's docs.

Contributor Author

MichaelKatsoulis Oct 25, 2021

yes I should add a readme there too


          Apply suggestions from code review

9cdb3e6

Co-authored-by: Chris Mark <chrismarkou92@gmail.com>

tetianakravchenko reviewed

View reviewed changes

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated

+              Step-by-step walkthrough
+. Kubernetes provider `init` function [adds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L46) the provider in the autodiscover providers registry at startup. For Kubernetes provider an `AutodiscoverBuilder` func is passed as an argument.
+. Metricbeat calls `NewAutodiscover` [function](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L183) which checks in the config for enabled providers and [builds](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/provider.go#L90) them one by one, calling the `AutodiscoverBuilder` func.
+. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) which is then added to an Autodiscover manager struct.

Contributor

tetianakravchenko Oct 21, 2021

https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131 points to the empty string now

libbeat/autodiscover/README.md Outdated

+. Kubernetes `AutodiscoverBuilder` creates and returns a [Kubernetes Provider struct](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L131) which is then added to an Autodiscover manager struct.
+. When unique is set to true [NewLeaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L141) is set as the eventManager of Kubernetes Provider.
+. Metricbeat [starts](https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L249) the Autodiscover manager which starts for Kubernets provider the [leaderElectionManager](https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L326). Before starting the providers it also starts a worker for listening of events that will be published by the eventers of each provider.
+. `OnStartedLeading` is executed when the specific metricbeat instance gains the leader election lock. [StartLeading](startLeading) cretaes a bus event with `"start":    true,`  and publishes it. The template configurations is also added in this event.

Contributor

tetianakravchenko Oct 21, 2021

should this link to startLeading work?
shouldn't it be https://github.com/elastic/beats/blob/master/libbeat/autodiscover/providers/kubernetes/kubernetes.go#L205 ?

Contributor Author

MichaelKatsoulis Oct 25, 2021

yeah! I don't know what happened there

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

libbeat/autodiscover/README.md Outdated Show resolved Hide resolved

MichaelKatsoulis and others added 4 commits

October 25, 2021 17:18


          Apply suggestions from code review

85df43b

Co-authored-by: Tetiana Kravchenko <tanya.kravchenko.v@gmail.com>


          Merge remote-tracking branch 'upstream/master' into k8s-autodiscover-…

217f2c3

…tech-doc


          Add readme for elastic agent kubernetes dynamic provider

aa264bd


          Update readme link

013f1d4

jsoriano added the Team:Integrations label

botelastic bot removed the needs_team label

botelastic bot commented Nov 28, 2021

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

botelastic bot added the Stalled label

botelastic bot commented Dec 28, 2021

Hi!
This PR has been stale for a while and we're going to close it as part of our cleanup procedure.
We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team.
Feel free to re-open this PR if you think it should stay open and is worth rebasing.
Thank you for your contribution!

botelastic bot closed this

ChrsMark reopened this

botelastic bot removed the Stalled label

ChrsMark assigned MichaelKatsoulis


          Merge remote-tracking branch 'upstream/master' into k8s-autodiscover-…

122ef18

…tech-doc


          Replace code links with permalinks

4e89dbb

Contributor Author

MichaelKatsoulis commented Jan 10, 2022

@ChrsMark @tetianakravchenko I updated all links of code in the READMEs with permalinks. I think it is safe, as at least the text will always correspond to the code.
Maybe master at some point will diverge a bit as we update the code.
I don't consider it to be a big issue. If we add a new feature or if we completely refactor a function/process then we should update the readmes as well.

tetianakravchenko approved these changes

View reviewed changes

MichaelKatsoulis merged commit cf3abb4 into elastic:master

v1v added a commit to v1v/beats that referenced this pull request


          Merge remote-tracking branch 'upstream/master' into feature/add-githu…

0bab1d5

…b-for-macos

* upstream/master: (172 commits)
  [Elastic Agent] Fix issue with ensureServiceToken. (elastic#29800)
  [Winlogbeat] Add provider name to Security routing pipeline check (elastic#29781)
  Add summary to journeys which don't emit journey:end (early node subprocess exits) (elastic#29606)
  Prepare 8.0.0-rc1 changelog (elastic#29795) (elastic#29806)
  Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29681)
  libbeat/processors/add_process_metadata: implement a process cache eviction policy (elastic#29717)
  [Automation] Update elastic stack version to 8.1.0-7004acda for testing (elastic#29783)
  Missing changelog entry for elastic#29773 (elastic#29791)
  Add a readme for k8s autodiscover provider (elastic#28213)
  Remove overriding of index pattern on the Kubernetes overview dashboard (elastic#29676)
  jjbb: remove obsoleted branches (<7.16) (elastic#29707)
  Add k8s metadata in state_cronjob metricset (elastic#29572)
  ibmmq: Fix timestamp parsing (elastic#29773)
  Do not add date to index if `@meta.index` is set (elastic#29775)
  ci: uses aliases for the branches (elastic#29706)
  Filebeat tests: Restore `@timestamp` field validation (elastic#29772)
  Forward port 7.16.3 changelog to master (elastic#29777)
  auditd: Store program arguments in process.args array (elastic#29601)
  System/socket: Support kernel_clone() replacement for _do_fork() (elastic#29744)
  Do not mention removal if version is not specified in `cfgwarn` messages (elastic#29727)
  ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-skip docs Team:Integrations