Reduce load on the Kubernetes API server and reduce the peak memory use of the cert-manager components by enabling the use of the WatchList (Streaming Lists) feature by wallrj · Pull Request #7175 · cert-manager/cert-manager

wallrj · 2024-07-12T16:53:32Z

Allow cert-manager users to enable the WatchListClient feature in client-go, so that they can experiment with the feature and evaluate its effect on the memory usage of the components.
They will also have to enable the WatchList feature of their Kubernetes API server.

This PR merges the Client-Go Feature Flags into the feature flags of each of the cert-manager components.
Copying the technique and copying some of the code used by the kube-controller-manager. See:
- Inject feature gate instance into client-go for kube components. kubernetes/kubernetes#122738
When the client-go WatchListClient feature is eventually promoted to GA, the feature flag will be hidden, but the flag will continue to work if enabled, but will error if disabled, because when GA, the it will no longer be possible to turn the implementation off.
I expect that new client-go feature flags will be introduced in future, which may or may not be applicable to cert-manager, but these will be disabled by default and should not affect cert-manager users.
We will need to be aware that the client-go features are being enabled in our E2E tests.
And we should update the documentation to explain which of the features are client-go features and link to the documentation for those features.
The ALPHA WatchList feature of the kube-apiserver is available since v1.27.
The ALPHA WatchListClient feature of client-go was introduced in v0.28.0, enabled with a non-standard ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environment variable.
The BETA WatchListClient feature was promoted in v0.31.0, with a new WatchListClient feature flag.
It is not clear when the WatchList and WatchListClient features will become GA (enabled by default).
The following comment from a K8S contributor says:

...but only after this feature graduates to GA, which most likely be not sooner than 1.33, although looking at 3157 it's still TBD.
I've measured a significant memory reduction in both the controller and in the kube-apiserver when the WatchList feature is enabled. (see testing section below).
There is no memory reduction in cainjector, because it only caches the metadata of Secrets.
There is no memory reduction in webhook, because it doesn't cache any resources.
I did not find any examples of other projects that have enabled this feature:
- https://github.com/search?q=%22WatchListClient%22+language%3AGo&type=code&l=Go

Fixes: #3748

Background

Release Note

Feature: Add a new `WatchListClient` feature flag to cert-manager controller, cainjector and webhook, which allows the components to use of the ALPHA `WatchList` / Streaming list feature of the Kubernetes API server. This reduces the load on  the Kubernetes API server when cert-manager starts up and reduces the peak memory usage in the cert-manager components.

Testing

Memory reduction in a Kind cluster

I deployed cert-manager from master and from this branch in a Kind cluster with 100MiB Secret resources.
Measured the peak memory usage of all controller, cainjector, webhook using the VmHWM file in /proc/*/status.

component	before	after
controller	362	229
cainjector	46	44
webhook	48	49
etcd	366	358
kube-apiserver	1134	824

See https://gist.github.com/wallrj/f15ad450f1b3effb107db5e6a01bf03f

Log messages

With --feature-gates=ClientWatchList=true and --v 6 you'll see the WATCH requests with the following query string parameters:

resourceVersionMatch=NotOlderThan
sendInitialEvents=true

kubectl logs -n cert-manager deploy/cert-manager | fgrep -v -e "starting worker" -e "leaderelection"

I0712 16:40:35.025446 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/secrets?allowWatchBookmarks=true&labelSelector=controller.cert-manager.io%2Ffao%3Dtrue&resourceVersionMatch=NotOlderThan&sendInitialEvents=true&timeout=7m37s&timeoutSeconds=457&watch=true 200 OK in 2502 milliseconds
I0712 16:40:35.025528 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/secrets?allowWatchBookmarks=true&labelSelector=%21controller.cert-manager.io%2Ffao&resourceVersionMatch=NotOlderThan&sendInitialEvents=true&timeout=8m7s&timeoutSeconds=487&watch=true 200 OK in 2501 milliseconds
I0712 16:40:35.655423 1 reflector.go:798] exiting k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232 Watch because received the bookmark that marks the end of initial events stream, total 6 items received in 3.13200872s
I0712 16:40:35.655514 1 reflector.go:359] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0712 16:40:35.655697 1 reflector.go:798] exiting k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232 Watch because received

Command Line Flags

You'll see the following new features among the feature flags help output.

$ go run ./cmd/controller --help | fgrep -A 20 -- --feature-gates
      --feature-gates mapStringBool                          A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
                                                             AdditionalCertificateOutputFormats=true|false (BETA - default=true)
                                                             AllAlpha=true|false (ALPHA - default=false)
                                                             AllBeta=true|false (BETA - default=false)
                                                             ExperimentalCertificateSigningRequestControllers=true|false (ALPHA - default=false)
                                                             ExperimentalGatewayAPISupport=true|false (BETA - default=true)
                                                             InformerResourceVersion=true|false (ALPHA - default=false)
                                                             LiteralCertificateSubject=true|false (BETA - default=true)
                                                             NameConstraints=true|false (ALPHA - default=false)
                                                             OtherNames=true|false (ALPHA - default=false)
                                                             SecretsFilteredCaching=true|false (BETA - default=true)
                                                             ServerSideApply=true|false (ALPHA - default=false)
                                                             StableCertificateRequestName=true|false (BETA - default=true)
                                                             UseCertificateRequestBasicConstraints=true|false (ALPHA - default=false)
                                                             UseDomainQualifiedFinalizer=true|false (ALPHA - default=false)
                                                             ValidateCAA=true|false (ALPHA - default=false)
                                                             WatchListClient=true|false (BETA - default=false)
...

$ go run ./cmd/cainjector --help

      --feature-gates mapStringBool                          A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
                                                             AllAlpha=true|false (ALPHA - default=false)
                                                             AllBeta=true|false (BETA - default=false)
                                                             InformerResourceVersion=true|false (ALPHA - default=false)
                                                             ServerSideApply=true|false (ALPHA - default=false)
                                                             WatchListClient=true|false (BETA - default=false)

$ go run ./cmd/webhook --help | fgrep -A 20 -- --feature-gates
      --feature-gates mapStringBool                          A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
                                                             AdditionalCertificateOutputFormats=true|false (BETA - default=true)
                                                             AllAlpha=true|false (ALPHA - default=false)
                                                             AllBeta=true|false (BETA - default=false)
                                                             InformerResourceVersion=true|false (ALPHA - default=false)
                                                             LiteralCertificateSubject=true|false (BETA - default=true)
                                                             NameConstraints=true|false (ALPHA - default=false)
                                                             OtherNames=true|false (ALPHA - default=false)
                                                             WatchListClient=true|false (BETA - default=false)

Testing cert-manager 1.15

I took the 1.15 binaries and enabled the client watch list feature using the old ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environment variable.
With 100MiB of Secrets

controller peak memory (VmHWM) dropped from 436 to 212 MB.
kube-apiserver peak memory dropped from 1134 to 772 MB.

https://gist.github.com/wallrj/c4933057037967d634708548e8aa7390

deploy/charts/cert-manager/templates/deployment.yaml

inteon · 2024-08-26T08:29:55Z

@wallrj we are now using the v0.31.0 client-go libraries.

jsoref · 2024-09-17T13:34:13Z

/kind bug

inteon · 2024-09-19T10:17:14Z

internal/controller/feature/features.go

+	// features are wired to the existing --feature-gates flag just as all other features
+	// are. Further, client-go features automatically support the existing mechanisms for
+	// feature enablement metrics and test overrides.
+	ca := &clientAdapter{utilfeature.DefaultMutableFeatureGate}


Maybe we should add this to cainjector and the webhook too?

Signed-off-by: Richard Wall <richard.wall@venafi.com>

wallrj

There are currently two client-go features:

https://github.com/kubernetes/kubernetes/blob/5973accf487eef985393cde200e7a4f9d09bc2e0/staging/src/k8s.io/client-go/features/known_features.go#L31-L43
InformerResourceVersion: Related to Storage Version Migration. See kubernetes/kubernetes#123344
WatchListClient: Related to streaming list feature. See kubernetes/kubernetes#122791

inteon

/approve
/lgtm

cert-manager-prow · 2024-09-24T13:18:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [inteon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wallrj · 2024-09-25T18:29:10Z

make/config/kind/cluster.yaml

+  #
+  # - https://kind.sigs.k8s.io/docs/user/configuration/#feature-gates
+  # - https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists
+  WatchList: true


This broke the K8S 1.25 and 1.26 E2E tests because the WatchList feature was added in 1.27.
We've decided to remove support for 1.25 and 1.26 in cert-manager 1.16.
See:

Update tested k8s versions to roughly match latest supported releases testing#1059

*not test, we might still support those versions

wallrj · 2024-09-26T06:27:17Z

/kind feature

wallrj commented Jul 17, 2024

View reviewed changes

deploy/charts/cert-manager/templates/deployment.yaml Outdated Show resolved Hide resolved

wallrj mentioned this pull request Jul 17, 2024

Cert-manager causes API server panic on clusters with more than 20000 secrets. #3748

Closed

wallrj changed the title ~~WIP: Enable the WatchList (Streaming Lists) feature~~ WIP: Avoid overloading the Kubernetes API server by enabling the WatchList (Streaming Lists) feature Jul 17, 2024

maelvls mentioned this pull request Jul 22, 2024

fix/vc-31703-agent-memory-startup-spikes jetstack/jetstack-secure#525

Merged

inteon changed the title ~~WIP: Avoid overloading the Kubernetes API server by enabling the WatchList (Streaming Lists) feature~~ WIP: Reduce load on the Kubernetes API server by enabling the WatchList (Streaming Lists) feature Aug 14, 2024

cert-manager-prow bot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 17, 2024

inteon self-assigned this Sep 19, 2024

inteon force-pushed the 3748-enable-watchlist-streaming-lists branch 2 times, most recently from cd44551 to 75f28b0 Compare September 19, 2024 10:12

inteon reviewed Sep 19, 2024

View reviewed changes

Enable the WatchList (Streaming Lists) feature

9ed80cf

Signed-off-by: Richard Wall <richard.wall@venafi.com>

wallrj force-pushed the 3748-enable-watchlist-streaming-lists branch from 75f28b0 to 9ed80cf Compare September 24, 2024 08:35

wallrj commented Sep 24, 2024

View reviewed changes

cert-manager-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 24, 2024

wallrj requested a review from inteon September 24, 2024 13:17

inteon reviewed Sep 24, 2024

View reviewed changes

cert-manager-prow bot added the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2024

cert-manager-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 24, 2024

cert-manager-prow bot merged commit 81bd1c5 into cert-manager:master Sep 24, 2024

wallrj deleted the 3748-enable-watchlist-streaming-lists branch September 24, 2024 13:20

wallrj commented Sep 25, 2024

View reviewed changes

cert-manager-prow bot added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce load on the Kubernetes API server and reduce the peak memory use of the cert-manager components by enabling the use of the WatchList (Streaming Lists) feature#7175

Reduce load on the Kubernetes API server and reduce the peak memory use of the cert-manager components by enabling the use of the WatchList (Streaming Lists) feature#7175
cert-manager-prow[bot] merged 1 commit intocert-manager:masterfrom
wallrj:3748-enable-watchlist-streaming-lists

wallrj commented Jul 12, 2024 •

edited

Loading

Uh oh!

Uh oh!

inteon commented Aug 26, 2024

Uh oh!

jsoref commented Sep 17, 2024

Uh oh!

inteon Sep 19, 2024

Uh oh!

wallrj left a comment

Uh oh!

inteon left a comment

Uh oh!

cert-manager-prow bot commented Sep 24, 2024

Uh oh!

wallrj Sep 25, 2024

Uh oh!

inteon Sep 25, 2024

Uh oh!

wallrj commented Sep 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wallrj commented Jul 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Release Note

Testing

Memory reduction in a Kind cluster

Log messages

Command Line Flags

Testing cert-manager 1.15

Uh oh!

Uh oh!

inteon commented Aug 26, 2024

Uh oh!

jsoref commented Sep 17, 2024

Uh oh!

inteon Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

wallrj left a comment

Choose a reason for hiding this comment

Uh oh!

inteon left a comment

Choose a reason for hiding this comment

Uh oh!

cert-manager-prow bot commented Sep 24, 2024

Uh oh!

wallrj Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

inteon Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

wallrj commented Sep 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wallrj commented Jul 12, 2024 •

edited

Loading