Add Multi-pool remaining IPs metric by rabelmervin · Pull Request #41397 · cilium/cilium

rabelmervin · 2025-08-27T08:55:05Z

Approach: Multi-Pool IPAM Remaining IPs Metrics

Overview
This PR implements real-time metrics tracking for remaining IP addresses in the multi-pool IPAM allocator, Which tracks remaining ips per pool

Implementation Details
Metrics : cilium_ipam_remaining_ips
Metrics Schema : Gauge

Registered metrics in start function present in allocator.go if metrics enabled.
Created new functions like GetAvailableAddrsV4(), GetAvailableAddrsV6() to compute available ips and allocatedV4(), allocatedV6() to compute allocated ips.
Implemented getRemainingips() to compute remaining ips using the above helpers and updates the metircs. RemainingIPS = availableIPS - allocatedIPS
Implemented CountAllocatedIPsInPrefix() in cidr_set.go to compute individual allocated ips in general.

pippolo84

Hi @rabelmervin . Thanks for the PR!

Could you please clarify your approach and maybe show an example of the new metrics actually running (maybe in a local kind cluster)?

I'm having some difficulties in understanding the code structure (e.g: where getRemainingIPs is called?).

rabelmervin · 2025-09-02T14:36:51Z

Hi @pippolo84 I have tested the changes several times with kind yet, the metric is not visible and also the existing trigger metrics like K8sSync, Resync

pippolo84 · 2025-09-04T15:44:13Z

Hi @pippolo84 I have tested the changes several times with kind yet, the metric is not visible and also the existing trigger metrics like K8sSync, Resync

Looking at the metrics for both cluster-pool and multi-pool IPAM mode, I realized that:

for cluster-pool, we have the k8s sync trigger metrics, but we do not register them, thus they are not exposed. I fixed that in ipam: Register k8s sync metrics for cluster-pool #41531
for multi-pool, we do not have any metrics (you can also see that from the fact that in *Allocator.Start the *metrics.Registry parameter is unused)

Therefore, to add metrics for multi-pool IPAM in the operator, you should:

create the needed metrics
register them to the registry passed in to the multi-pool allocator (if enable-metrics is true for the operator)

You can take a look at #41531 to see how you can look at the metrics exposed by a running operator.

Depending on the metrics you want to add, you might need to add them in the agent (e.g: how many IPs are used for a node). The steps needed to do that should be the same anyway.

maintainer-s-little-helper · 2025-09-16T13:27:36Z

Commit bbd7aa3 does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

rabelmervin · 2025-09-16T14:40:41Z

PR has been closed due to deletion of root commit and local errors. made a new pr HERE

tklauser · 2025-09-17T08:09:41Z

@rabelmervin you can rebase your PR branch on latest main. There should be no need to close the PR and create a new one. Let's keep the discussion in this PR.

maintainer-s-little-helper · 2025-09-17T08:11:36Z

Commit bbd7aa3 does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

rabelmervin · 2025-09-17T11:33:13Z

@rabelmervin you can rebase your PR branch on latest main. There should be no need to close the PR and create a new one. Let's keep the discussion in this PR.

It worked thanks for your feedback @tklauser could you please review the changes?

tklauser · 2025-09-17T11:57:48Z

@rabelmervin could you please add some description to the PR and the commit message explaining the approach (as requested by @pippolo84 in #41397 (review)).

Also, please describe how you tested this and ideally also add a unit test to verify that the metric is working as intended.

pippolo84

Differently from cluster-pool, here we are not adding metrics tied to a trigger, we just want to update a metric that keeps track of the available IPs for each pool.
I suggest to rework the code without relying too much on the cluster-pool implementation. You can consider it as an inspiration to see how the metrics are registered, but the logic to update it will inevitably be different.

pkg/ipam/allocator/multipool/allocator.go

pippolo84 · 2025-09-19T14:18:25Z

pkg/ipam/allocator/multipool/pool_allocator.go

+	nodes                     map[string]poolToCIDRs // nodeName -> pool -> cidrs
+	ready                     bool
+	metricsAPI                metricsAPI
+	CountAllocatedIPsInPrefix func(cidr *netip.Prefix) *big.Int


Not sure why we need a function as a field of this struct. What is the reason behind this?

Helper functions using it (allocatedv4 and v6) to calculate the allocated ips

But why a field in a struct and not a simple function in the multipool package?

Also, CountAllocatedIPsInPrefix is initialized in NewPoolAllocator to always returns 0 and never overwritten, so it will always return 0. What's the point?

pkg/ipam/metrics/metrics.go

Signed-off-by: rabelmervin <rabelmervin@gmail.com> Added Prometheus metrics for multipool IPAM remaining IPs Add metric for remaining IPs in multi-pool setup Introduce a metric to track the remaining IPs available in a multi-pool environment. This enhancement enables better monitoring and resource management in scenarios where multiple IP pools are utilized. - Provides visibility into IP usage across all pools - Ensures proactive alerting for potential IP exhaustion - Aligns with the existing observability features Fixes cilium#41397

Signed-off-by: rabelmervin <rabelmervin@gmail.com>

pchaigno · 2025-11-24T15:19:24Z

@rabelmervin It seems there's at least one question from Fabio left answered above.

pchaigno · 2025-11-24T15:19:32Z

/test

pippolo84

Multi-Pool IPAM allocates IP addresses in two steps. Consider that the pool 1.1.1.0/24 is available from a CiliumPodIPPool resource.

The operator divide that CIDR into smaller CIDRs with the same mask size that will be allocated to each agent. As an example, say that mask size is equal to 28, then the operator will allocate 1.1.1.0/28, then 1.1.1.16/28 and so on.
Each agent allocates single IP addresses to the scheduled pods on the node. Following the example, say that a node has been assigned the CIDR 1.1.1.0/28 from the operator: it will assign to the pods the IP addresses 1.1.1.1, then 1.1.1.2 and so on.

If we want to add metrics for remaining IPs, the first question we should ask is what are we going to expose in the metrics. One possible approach is:

adding metrics in the operator to expose how many CIDRs of size maskSize we can still allocate to nodes from all the CIDRs in each pools.
adding metrics in the agents to expose how many IP addresses we can still allocate from the CIDRs of size maskSize that have been assigned to the node.

I suggest to start from the operator metrics and leave 2) for a follow up PR.

pkg/ipam/allocator/multipool/pool_allocator.go

pippolo84 · 2025-12-10T10:46:37Z

pkg/ipam/allocator/multipool/pool_allocator_test.go

+	allocatedV4 := pool.allocatedV4()
+	assert.NotNil(t, allocatedV4)
+	assert.True(t, allocatedV4.Cmp(big.NewInt(0)) >= 0, "Allocated IPv4 should be non-negative")
+
+	// Test allocatedV6()
+	allocatedV6 := pool.allocatedV6()
+	assert.NotNil(t, allocatedV6)
+	assert.True(t, allocatedV6.Cmp(big.NewInt(0)) >= 0, "Allocated IPv6 should be non-negative")


To make the test meaningful we should try to allocate some CIDRs and verify that the numbers of allocated and avalaible IPs are the ones expected.

pippolo84 · 2025-12-10T10:47:02Z

pkg/ipam/allocator/multipool/pool_allocator_test.go

+	remainingV4 := new(big.Int).Sub(availableV4, allocatedV4)
+	remainingV6 := new(big.Int).Sub(availableV6, allocatedV6)
+
+	assert.True(t, remainingV4.Cmp(big.NewInt(0)) >= 0, "Remaining IPv4 should be non-negative")
+	assert.True(t, remainingV6.Cmp(big.NewInt(0)) >= 0, "Remaining IPv6 should be non-negative")


What is the purpose of testing big.Int.Sub?

maintainer-s-little-helper · 2025-12-14T07:25:32Z

Commit 22ca075 does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

pchaigno · 2026-02-11T13:04:08Z

Switching to draft while the review is being addressed.

github-actions · 2026-03-14T02:14:12Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

github-actions · 2026-03-29T02:14:34Z

This pull request has not seen any activity since it was marked stale.
Closing.

rabelmervin requested a review from a team as a code owner August 27, 2025 08:55

rabelmervin requested a review from pippolo84 August 27, 2025 08:55

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Aug 27, 2025

github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Aug 27, 2025

pippolo84 requested changes Aug 29, 2025

View reviewed changes

rabelmervin force-pushed the multipool branch from fdce6dd to 3f6e396 Compare August 29, 2025 16:32

rabelmervin requested a review from pippolo84 August 30, 2025 09:55

ldelossa added area/metrics Impacts statistics / metrics gathering, eg via Prometheus. area/ipam IP address management, including cloud IPAM release-note/misc This PR makes changes that have no direct user impact. labels Sep 2, 2025

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Sep 2, 2025

rabelmervin force-pushed the multipool branch from 3f6e396 to e2a099a Compare September 9, 2025 12:20

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Sep 16, 2025

rabelmervin closed this Sep 16, 2025

tklauser mentioned this pull request Sep 17, 2025

Implementation of Multi-pool remaining IPs metric #41706

Closed

tklauser reopened this Sep 17, 2025

rabelmervin force-pushed the multipool branch from bbd7aa3 to 70d71de Compare September 17, 2025 11:21

maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Sep 17, 2025

pippolo84 requested changes Sep 19, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Oct 5, 2025

rabelmervin force-pushed the multipool branch from 5fa4e38 to df7704b Compare October 5, 2025 13:40

rabelmervin marked this pull request as draft October 23, 2025 12:01

This comment was marked as resolved.

Sign in to view

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Nov 1, 2025

rabelmervin force-pushed the multipool branch from c375c5e to 90a0d69 Compare November 1, 2025 14:27

maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Nov 1, 2025

rabelmervin force-pushed the multipool branch from 90a0d69 to 50c4e47 Compare November 1, 2025 14:32

rabelmervin marked this pull request as ready for review November 4, 2025 16:52

rabelmervin marked this pull request as draft November 4, 2025 17:39

Add Prometheus metrics for multipool IPAM remaining IPs

dd601a7

Signed-off-by: rabelmervin <rabelmervin@gmail.com>

rabelmervin force-pushed the multipool branch from a2a2e8f to dd601a7 Compare November 5, 2025 03:01

rabelmervin marked this pull request as ready for review November 5, 2025 03:02

pchaigno requested a review from pippolo84 November 24, 2025 15:18

pippolo84 requested changes Dec 10, 2025

View reviewed changes

Merge branch 'cilium:main' into multipool

22ca075

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Dec 14, 2025

joestringer changed the title ~~Implementation of Multi-pool remaining IPs metric~~ Add Multi-pool remaining IPs metric Jan 9, 2026

aanm removed the dont-merge/wait-until-release Freeze window for current release is blocking non-bugfix PRs label Jan 14, 2026

pchaigno marked this pull request as draft February 11, 2026 13:04

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Mar 14, 2026

github-actions bot closed this Mar 29, 2026

Conversation

rabelmervin commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approach: Multi-Pool IPAM Remaining IPs Metrics

Uh oh!

pippolo84 left a comment

Choose a reason for hiding this comment

Uh oh!

rabelmervin commented Sep 2, 2025

Uh oh!

pippolo84 commented Sep 4, 2025

Uh oh!

maintainer-s-little-helper bot commented Sep 16, 2025

Uh oh!

rabelmervin commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tklauser commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maintainer-s-little-helper bot commented Sep 17, 2025

Uh oh!

rabelmervin commented Sep 17, 2025

Uh oh!

tklauser commented Sep 17, 2025

Uh oh!

pippolo84 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pippolo84 Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

rabelmervin Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pippolo84 Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

pchaigno commented Nov 24, 2025

Uh oh!

pchaigno commented Nov 24, 2025

Uh oh!

pippolo84 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pippolo84 Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

pippolo84 Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

maintainer-s-little-helper bot commented Dec 14, 2025

Uh oh!

pchaigno commented Feb 11, 2026

Uh oh!

github-actions bot commented Mar 14, 2026

Uh oh!

github-actions bot commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

rabelmervin commented Aug 27, 2025 •

edited

Loading

rabelmervin commented Sep 16, 2025 •

edited

Loading

tklauser commented Sep 17, 2025 •

edited

Loading

rabelmervin Oct 8, 2025 •

edited

Loading