Skip to content

Add metadata watcher and informer#111

Merged
swiatekm merged 3 commits intomainfrom
feat/metadatawatcher
Oct 3, 2024
Merged

Add metadata watcher and informer#111
swiatekm merged 3 commits intomainfrom
feat/metadatawatcher

Conversation

@swiatekm
Copy link
Copy Markdown
Member

@swiatekm swiatekm commented Oct 2, 2024

Introduce metadata-only watchers to the kubernetes package. These are useful if we only need to track metadata for a resource - a good example are ReplicaSets, for which we usually only care about the OwnerReferences. As a result, we only store the metadata, reducing steady-state memory consumption, but also only get updates involving metadata, reducing churn greatly in larger clusters.

The implementation introduces new constructors for the Watcher, allowing an informer to be passed in. Existing constructors are implemented using the new constructor, though none of the code actually changes. As a result, it is now possible to unit test the watcher, and I've added some basic unit tests for it.

We also add two helper functions:

  • GetKubernetesMetadataClient creates a metadata-only kubernetes client, and is very similar to the existing GetKubernetesClient
  • RemoveUnnecessaryReplicaSetData is a transform function that can be passed into an informer so it only stores the metadata we actually use

I tested these new functions in both beats and agent, in a kind cluster as well as one of our staging clusters.

This is part of the solution to elastic/elastic-agent#5580.

@swiatekm swiatekm requested a review from a team as a code owner October 2, 2024 13:16
@swiatekm swiatekm force-pushed the feat/metadatawatcher branch 3 times, most recently from 64b342a to 962b390 Compare October 2, 2024 13:39
@swiatekm swiatekm force-pushed the feat/metadatawatcher branch from 962b390 to 3fe2e43 Compare October 2, 2024 14:10
@ycombinator ycombinator added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Oct 2, 2024
@pierrehilbert pierrehilbert requested review from mauri870 and removed request for khushijain21 October 3, 2024 07:44
@swiatekm swiatekm force-pushed the feat/metadatawatcher branch from 3fe2e43 to d7f0a73 Compare October 3, 2024 09:40
Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
@swiatekm swiatekm requested a review from mauri870 October 3, 2024 11:27
Copy link
Copy Markdown
Member

@mauri870 mauri870 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

var cachedObject runtime.Object

store = informer.GetStore()
queue = workqueue.NewNamed(name)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
queue = workqueue.NewNamed(name)
queue = workqueue.NewWithConfig(QueueConfig{
Name: name,
})

nit: since you touch this code maybe you wanna deal with deprecated calls as well

Copy link
Copy Markdown
Member Author

@swiatekm swiatekm Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually want to avoid touching it as much as possible. I think the recommended way here is to use typed queues, and I don't want to make that kind of change in this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@swiatekm could you make an issue or PR (if it's quicker but I doubt that) to address the deprecated calls? That way we don't lose track of this work. Thanks.

Copy link
Copy Markdown

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside some nitpicking this LGTM

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

History

@swiatekm swiatekm merged commit a698e0f into main Oct 3, 2024
@swiatekm swiatekm deleted the feat/metadatawatcher branch October 3, 2024 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants