Auto-generate feature metrics documentation for agent and operator by ajmmm · Pull Request #44556 · cilium/cilium

ajmmm · 2026-02-27T15:46:02Z

Following #44114 CI will now fail if feature metric documentation is out of sync. However, as noted, the developer U/X for this is poor.

This PR:

Introduces a new kind install target: make kind-install-cilium-metrics.
This can be run to install Cilium with the necessary config to express default metrics. This is required to auto-generate the documentation.
Introduces a new docs target: make -C Documentation update-metrics.
This queries the installed kind cluster and auto-generates the feature metric files in Documentation/observability.
Updates the tests-smoke.yaml file to better express any detected sync issues for these files, and also mentions the new build targets that can be used to re-generate them from command line.
The standard make -C Documentation check is also updated to check the metrics files.

Note: the smoke test workflow is now duplicating some logic, but thought it best to leave this alone for now until these changes are accepted. I envisage a follow-up PR in due course that could split the testing out into its own workflow and refactoring to reducing duplication.

CI Test example

Screenshot below from test CI run with a commit that modifies one of the documents.

Introduce build targets to resynchronise feature metric documentation files for the agent and operator.

Introduce a new make target that installs Cilium with the necessary configuration and environment variables to export default metrics. This will enable us to programmatically generate feature metrics documentation in the next commit. Signed-off-by: Alasdair McWilliam <alasdair.mcwilliam@isovalent.com>

Introduce a new update-metrics build target in Documentation. When combined with installing Cilium via `make kind-install-cilium-metrics`, this provides a mechanism to automatically generate feature metrics documentation for the agent and operator. This commit also adds Documentation/check-metrics.sh which is now called via `make -C Documentation check`. Signed-off-by: Alasdair McWilliam <alasdair.mcwilliam@isovalent.com>

Commit 677f909 ("gh: fix metric updates in smoke test workflow") updated the IPv4 smoke test to fail if feature metrics are out of sync. However, when hitting this in CI, it's not obvious how to go about fixing this. The previous two commits have provided a simplified way to: 1. Install Cilium with the correct configuration and environment variables to export default metrics. This is: `make kind-install-cilium-metrics` 2. Query the installed kind cluster for feature metrics and process these using the included tooling to produce the relevant docs for the agent and operator. This is: `make -C Documentation update-metrics` Therefore, this commit updates the smoke test to include instructions on how to fix this issue. Fixes: 677f909 ("gh: fix metric updates in smoke test workflow") Signed-off-by: Alasdair McWilliam <alasdair.mcwilliam@isovalent.com>

ajmmm · 2026-02-27T16:24:59Z

/test

qmonnet

Looks OK but I'm concerned with the readability and maintainability of the Makefile recipe from the second commit.

Ideally, these changes should also come with an update to Documentation/contributing/docs/docsframework.rst 🙂

qmonnet · 2026-02-27T17:09:56Z

Documentation/Makefile

+update-metrics: OPERATOR_METRICS_DOC := ./observability/feature-metrics-operator.txt
+update-metrics: OPERATOR_METRICS_PREFIX := cilium_operator_feature
+update-metrics: OPERATOR_METRICS_SEPARATORS := adv_connect_and_lb
+update-metrics:


I find this whole Makefile recipe rather difficult to read. Have you considered moving it to a dedicated script?

+1 would be good to follow the current pattern for Documentation/update-*

I did actually start with an update script, but I considered the need to then re-acquire variables from the Makefile. Combined with needing to then run promtool via docker, it all got a bit messy. (That was admittedly before the 8 make variables that appeared!)

Ack, will refactor into a script :)

joestringer

Thanks for working on keeping these metrics docs up to date btw. I brought up some thoughts below on how to reduce toil for contributors, but we can probably tackle this in steps. Some minor feedback below to address in this PR though.

joestringer · 2026-02-27T18:31:02Z

Makefile.kind

 		--version=

+.PHONY: kind-install-cilium-metrics
+kind-install-cilium-metrics: check_deps kind-ready


Please also add a comment at the end for help text. You can see how it shows up using make help | grep install.

joestringer · 2026-02-27T18:32:18Z

Makefile.kind

+	$(CILIUM_CLI) install \
+		--chart-directory=$(ROOT_DIR)/install/kubernetes/cilium \
+		$(KIND_VALUES_FILES) \
+		--helm-values=$(ROOT_DIR)/contrib/testing/kind-metrics.yaml \


Actually, is this something we should just integrate into an existing target? I see this is a new target but it's mostly just adding the extra metrics. I don't see this new target being used anywhere. Maybe we can collectively as a developer community be more consistent about metrics if the default dev install targets also configure the metrics(?)

Yeah, I did think about just adding them to the base kind-common.yaml. But then I didn't want to disrupt the status quo of everyone else who uses make kind*. I can just make it default if it's cleaner. :)

Do you know if there's anything inconvenient about enabling this by default? If yes, that could be a reason to keep it separate. If no, I'd be somewhat inclined to just update it for everyone so all developers see the same thing. If someone complains, we can reopen the discussion.

I can't think of anything much, but that doesn't mean there isn't an impact! But, it's a bit moot, I've got a replacement PR coming in.

joestringer · 2026-02-27T18:33:10Z

Documentation/Makefile

+update-metrics: OPERATOR_METRICS_DOC := ./observability/feature-metrics-operator.txt
+update-metrics: OPERATOR_METRICS_PREFIX := cilium_operator_feature
+update-metrics: OPERATOR_METRICS_SEPARATORS := adv_connect_and_lb
+update-metrics:


+1 would be good to follow the current pattern for Documentation/update-*

joestringer · 2026-02-27T18:35:29Z

Documentation/Makefile

 		--env INCREMENTAL=$(INCREMENTAL)
 DOCKER_RUN := $(DOCKER_CTR) $(DOCS_BUILDER_IMG)

+PROMTOOL_IMG ?= quay.io/prometheus/prometheus:latest


Let's pin this to a specific sha and let renovate handle versioning. We don't want to arbitrarily break the tree because prometheus releases a new version of their linter.

nit, not a requirement for this PR, but .github/workflows/tests-smoke.yaml also pulls in promtool, maybe that could be rebased to invoke the same check rather than open-coding the install & use of promtool in the GitHub workflow.

joestringer · 2026-02-27T18:37:56Z

Documentation/check-metrics.sh

+
+if ! git diff --quiet -- "${metrics_dir}" ; then
+    git --no-pager diff "${metrics_dir}"
+    echo "HINT: to fix this, install via 'make kind-install-cilium-metrics' and run 'make -C Documentation update-metrics'"


nit: Idly wondering if we could have a model more similar to cmdref where we don't have to run a full cluster & docker image in order to test, but if we could instead just build cells with the metrics related code and then invoke a new entrypoint with the metrics configuration in order to validate. I'm not sure how much work this would be but it might make it easier for us to just expect someone to run make -C Documentation update-metrics, have that do a similar dependency graph to update-cmdref and directly invoke static binaries rather than requiring the contributor to do extra steps & setup

It would be nicer if this can just be spat out of the build somehow, but I did it as-is just to get something in quickly. If the environment variable(s) are instead added to kind-common, in the interim, I just echo the variable that needs to be set and the make -C ...? Then in future this can be updated to remove the dependency on the env variable being set.

ajmmm · 2026-03-10T18:40:44Z

Because I've reworked how this is done to better align to the cmdref way of working, I've got a V2 PR that supersedes this now: #44715

I will close this.

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Feb 27, 2026

ajmmm force-pushed the pr/auto-generate-feature-metrics branch from 510b9a1 to 3acf103 Compare February 27, 2026 16:19

ajmmm added 3 commits February 27, 2026 16:22

ajmmm force-pushed the pr/auto-generate-feature-metrics branch from 3acf103 to 2d492fe Compare February 27, 2026 16:22

ajmmm mentioned this pull request Feb 27, 2026

Revert: Resync feature metrics documentation #44535

Closed

ajmmm marked this pull request as ready for review February 27, 2026 16:42

ajmmm requested review from a team as code owners February 27, 2026 16:42

ajmmm requested review from brlbil, qmonnet and thorn3r February 27, 2026 16:42

qmonnet reviewed Feb 27, 2026

View reviewed changes

joestringer reviewed Feb 27, 2026

View reviewed changes

joestringer added the release-note/misc This PR makes changes that have no direct user impact. label Feb 27, 2026

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Feb 27, 2026

ajmmm closed this Mar 10, 2026

ajmmm mentioned this pull request Mar 10, 2026

Auto-generate feature metrics documentation for agent and operator - v2 #44715

Merged

Conversation

ajmmm commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajmmm commented Feb 27, 2026

Uh oh!

qmonnet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joestringer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajmmm Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajmmm Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajmmm commented Mar 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ajmmm commented Feb 27, 2026 •

edited

Loading

joestringer left a comment •

edited

Loading

ajmmm Mar 9, 2026 •

edited

Loading

ajmmm Mar 9, 2026 •

edited

Loading