Add kubernetes_leaderelection provider by ChrsMark · Pull Request #24913 · elastic/beats

ChrsMark · 2021-04-02T11:44:39Z

What does this PR do?

Adds support for leaderelection provider in Agent based on #24267.

After this one is merged I plan to tune standalone manifest in follow-up PR so as to remove deployment section and only deploy as Deamonset with leaderelection "on".

Why is it important?

To support cluster wide metrics collection by only one Agent at a time between a group of multiple Agents (ie Deamonset).

On a second step we can change the standalone manifest, completely remove the Deployment of Agent and go on with only the Daemonset leveraging the leaderelection feature.

How to test this PR locally

Enable the provider and an input based on the leader condition:

providers:
  kubernetes_leaderelection:
    kube_config: /Users/chrismark/.kube/config
    leader_lease: elastic-agent
  
inputs:
  - type: logfile
    condition: ${kubernetes_leaderelection.leader} == true
    streams:
      - paths: /var/log/leaderlog/another.log

run inspect command to verify the output: ./elastic-agent -e -d "*" -c /elastic-agent.yml inspect output -o default
Verify the conditional input is enabled:

filebeat:
  inputs:
  - index: logs-generic-default
    paths: /var/log/leaderlog/another.log
    processors:
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    - add_fields:
        fields:
          id: 32806c0a-4f04-499c-9427-e6e24e5f6035
          snapshot: false
          version: 8.0.0
        target: elastic_agent
    type: log

(Optional:) Perform same test from different Agents and on a running mode so as to verify that only one of them picks the lock and becomes the leader

Related issues

Closes [Agent] Implement leader election on k8s for Elastic Agent #24267

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-04-02T11:44:40Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-04-02T11:44:41Z

Pinging @elastic/integrations (Team:Integrations)

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-04-02T11:49:06Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by user Chris Mark
Start Time: 2021-04-07T14:12:19.936+0000
Duration: 79 min 26 sec
Commit: 8f84f23

Test stats 🧪

Test	Results
Failed	0
Passed	6612
Skipped	16
Total	6628

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6612
Skipped	16
Total	6628

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark · 2021-04-06T13:57:14Z

@blakerouse could you have a look into this one please when you have the time? I mostly want to get feedback from Agent's team point of view.

deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role.yaml

...elastic-agent/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go

blakerouse · 2021-04-06T18:09:47Z

x-pack/elastic-agent/pkg/composable/providers/kubernetesleaderelection/config.go

+type Config struct {
+	KubeConfig string `config:"kube_config"`
+	// Name of the leaderelection lease
+	LeaderLease string `config:"leader_lease"`


So by default running Elastic Agent this is off. Is that the behavior we want? Being that will require a specific setting in the providers top-level key, and not something that could then be used by Fleet at the moment.

I think that we should have this enabled by default, by having it have a default value.

Agree, I will add it.

blakerouse · 2021-04-06T18:10:59Z

...elastic-agent/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go

+		id = "elastic-agent-leader-" + agentInfo.AgentID()
+	}
+
+	ns, err := kubernetes.InClusterNamespace()


Should there also be a configuration value for this? Might want to run it on a different namespace?

The convention here is that we will create the lease object under the same namespace with one where Agent is running. We also add role so as Agent to have access only to leases under the same namespace, see #24913 (comment)

So I think we should keep it as is and not expose this as an option to the user.

...elastic-agent/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go

blakerouse · 2021-04-06T18:13:24Z

...elastic-agent/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go

+)
+
+func init() {
+	composable.Providers.AddDynamicProvider("kubernetes_leaderelection", DynamicProviderBuilder)


I question if this should be a dynamic provider. I think this should be a context provider. That way leader election can effect all the other vars even dynamic ones discovered by the main kubernetes provider.

👍🏼 ok I can change it

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark · 2021-04-07T11:11:45Z

@jsoriano @blakerouse thanks for your review/comments. I think have I covered your comments so you can consider it ready for another review round.

blakerouse

Looks great!

blakerouse · 2021-04-07T12:51:45Z

...elastic-agent/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go

+		id = "elastic-agent-leader-" + agentInfo.AgentID()
+	}
+
+	ns, err := kubernetes.InClusterNamespace()


jsoriano

👍

(cherry picked from commit 2494ae5)

…4984)

Add kubernetes_leaderelection provider

ec6b00d

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added Team:Integrations Label for the Integrations team v7.13.0 Team:Elastic-Agent Label for the Agent team labels Apr 2, 2021

ChrsMark requested review from blakerouse, exekias and jsoriano April 2, 2021 11:44

ChrsMark self-assigned this Apr 2, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 2, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 2, 2021

ChrsMark added 3 commits April 2, 2021 14:45

fix config

a9cac2e

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fix comment

b898bca

Signed-off-by: chrismark <chrismarkou92@gmail.com>

remove out of scope fix

b6df824

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Fix leader holder identification

d3f9046

Signed-off-by: chrismark <chrismarkou92@gmail.com>

jsoriano reviewed Apr 6, 2021

View reviewed changes

blakerouse requested changes Apr 6, 2021

View reviewed changes

ChrsMark added 6 commits April 7, 2021 13:39

Fix k8s roles

511cde3

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Enable leaderelection by default and remove fields

fd630f7

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Make provider of type ContextProvider instead of DynamicProvider

04aac36

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fix leftover role

69ea5b7

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fix leftover role

1c1a4b9

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fix leftover role

8f84f23

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark requested review from blakerouse and jsoriano April 7, 2021 11:11

blakerouse approved these changes Apr 7, 2021

View reviewed changes

jsoriano mentioned this pull request Apr 7, 2021

Update k8s manifests to use proper roles' scope for leaderelection #24958

Merged

jsoriano approved these changes Apr 7, 2021

View reviewed changes

ChrsMark merged commit 2494ae5 into elastic:master Apr 8, 2021

ChrsMark added a commit to ChrsMark/beats that referenced this pull request Apr 8, 2021

Add kubernetes_leaderelection provider (elastic#24913)

00ac945

(cherry picked from commit 2494ae5)

ChrsMark added a commit to ChrsMark/beats that referenced this pull request Apr 8, 2021

Add kubernetes_leaderelection provider (elastic#24913)

cc26357

(cherry picked from commit 2494ae5)

This was referenced Apr 8, 2021

Cherry-pick #24913 to 7.x: Add kubernetes_leaderelection provider #24984

Merged

Add changelog entry for leaderelection on Agent #24985

Merged

Add leaderelection docs elastic/observability-docs#499

Merged

ChrsMark added a commit that referenced this pull request Apr 8, 2021

Cherry-pick #24913 to 7.x: Add kubernetes_leaderelection provider (#2…

dcd6b22

…4984)

This was referenced Apr 12, 2021

Leverege leader election in agent k8s manifests #25016

Merged

Cherry-pick #25016 to 7.x: Leverege leader election in agent k8s manifests #25033

Merged

Conversation

ChrsMark commented Apr 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Apr 2, 2021

Uh oh!

elasticmachine commented Apr 2, 2021

Uh oh!

elasticmachine commented Apr 2, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

ChrsMark commented Apr 6, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrsMark commented Apr 7, 2021

Uh oh!

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChrsMark commented Apr 2, 2021 •

edited

Loading

elasticmachine commented Apr 2, 2021 •

edited by jenkins-beats-ci bot

Loading