CI: Conformance Multi Pool IPAM (ci-multi-pool) #44107
Description
CI failure
Test [check-log-errors]:
🟥 check-log-errors/no-errors-in-logs:kind-chart-testing/kube-system/cilium-shvqm (cilium-agent): Non-zero (1) restart count of kind-chart-testing/kube-system/cilium-shvqm (cilium-agent) must be investigated
cilium-sysdump-Direct Routing-20260201-194905.zip
https://github.com/cilium/cilium/actions/runs/21568844069
2026-02-01T19:39:22.874872157Z time=2026-02-01T19:39:22.874629085Z level=debug source=/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:184 msg="Timer job triggered" module=agent.controlplane.ipam name=multi-pool-cilium-node-updater func=ipam.(*multiPoolManager).updateLocalNode
2026-02-01T19:39:22.874885832Z time=2026-02-01T19:39:22.874700147Z level=debug source=/go/src/github.com/cilium/cilium/pkg/ipam/pool.go:206 msg="releasing CIDR" module=agent.controlplane.ipam cidr=fd00::140/122
2026-02-01T19:39:22.874843423Z fatal error: concurrent map iteration and map write
2026-02-01T19:39:22.877967109Z
2026-02-01T19:39:22.877976938Z goroutine 5031 [running]:
2026-02-01T19:39:22.877980334Z internal/runtime/maps.fatal({0x4b9f3b0?, 0xc000e98690?})
2026-02-01T19:39:22.877983209Z /usr/local/go/src/runtime/panic.go:1046 +0x18
2026-02-01T19:39:22.877985814Z internal/runtime/maps.(*Iter).Next(0xc0030e1228?)
2026-02-01T19:39:22.877989601Z /usr/local/go/src/internal/runtime/maps/table.go:792 +0x86
2026-02-01T19:39:22.877992126Z github.com/cilium/cilium/pkg/ipam.(*multiPoolAllocator).Capacity(0xc0012a1398)
2026-02-01T19:39:22.877999049Z /go/src/github.com/cilium/cilium/pkg/ipam/multipool.go:120 +0x78
2026-02-01T19:39:22.878001924Z github.com/cilium/cilium/pkg/ipam.(*IPAM).allocateNextFamily(0xc001657e60, {0x4af9aaa, 0x4}, {0xc0022c5380, 0x2e}, {0x0, 0x0}, 0x1)
2026-02-01T19:39:22.878017233Z /go/src/github.com/cilium/cilium/pkg/ipam/allocator.go:136 +0x12b
2026-02-01T19:39:22.878022993Z github.com/cilium/cilium/pkg/ipam.(*IPAM).AllocateNextFamily(0xc001657e60, {0x4af9aaa, 0x4}, {0xc0022c5380, 0x2e}, {0x0, 0x0})
2026-02-01T19:39:22.878028674Z /go/src/github.com/cilium/cilium/pkg/ipam/allocator.go:200 +0xe5
2026-02-01T19:39:22.878043311Z github.com/cilium/cilium/pkg/ipam.(*IPAM).AllocateNext(0xc001657e60, {0x0, 0x0}, {0xc0022c5380, 0x2e}, {0x0, 0x0})
2026-02-01T19:39:22.878049442Z /go/src/github.com/cilium/cilium/pkg/ipam/allocator.go:220 +0xa5
2026-02-01T19:39:22.878101780Z github.com/cilium/cilium/pkg/ipam.(*IPAM).AllocateNextWithExpiration(0xc001657e60, {0x0?, 0xc000fd0990?}, {0xc0022c5380?, 0x47f005?}, {0x0?, 0x4693f20?}, 0x8bb2c97000)
2026-02-01T19:39:22.878107190Z /go/src/github.com/cilium/cilium/pkg/ipam/allocator.go:244 +0x3f
2026-02-01T19:39:22.878112900Z github.com/cilium/cilium/pkg/ipam/api.(*IpamPostIpamHandler).Handle(0xc000c5e020, {0xc0005c9cc0, 0xc000d0a689, 0x0, 0xc0010dfce0, 0x0})
2026-02-01T19:39:22.878115585Z /go/src/github.com/cilium/cilium/pkg/ipam/api/ipam_api_handler.go:52 +0xe7
2026-02-01T19:39:22.878118350Z github.com/cilium/cilium/api/v1/server/restapi/ipam.(*PostIpam).ServeHTTP(0xc000b3f848, {0x52208b0, 0xc0057998a8}, 0xc0005c9cc0)
2026-02-01T19:39:22.878121246Z /go/src/github.com/cilium/cilium/api/v1/server/restapi/ipam/post_ipam.go:56 +0x15d
2026-02-01T19:39:22.878212405Z github.com/go-openapi/runtime/middleware.(*Context).RoutesHandler.NewOperationExecutor.func1({0x52208b0, 0xc0057998a8}, 0xc0005c9cc0)
2026-02-01T19:39:22.878216854Z /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/operation.go:17 +0x50
2026-02-01T19:39:22.878220560Z net/http.HandlerFunc.ServeHTTP(0xc00103b500?, {0x52208b0?, 0xc0057998a8?}, 0xc0030e1770?)
2026-02-01T19:39:22.878223696Z /usr/local/go/src/net/http/server.go:2322 +0x29
2026-02-01T19:39:22.878226311Z github.com/go-openapi/runtime/middleware.NewRouter.func1({0x52208b0, 0xc0057998a8}, 0xc001c05e00)
2026-02-01T19:39:22.878228946Z /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/router.go:67 +0x257
2026-02-01T19:39:22.878231461Z net/http.HandlerFunc.ServeHTTP(0xc003489045?, {0x52208b0?, 0xc0057998a8?}, 0xc000fd0900?)
2026-02-01T19:39:22.878234497Z /usr/local/go/src/net/http/server.go:2322 +0x29
2026-02-01T19:39:22.878237723Z github.com/go-openapi/runtime/middleware.Redoc.serveUI.func1({0x52208b0, 0xc0057998a8}, 0xc001c05e00)
2026-02-01T19:39:22.878263039Z /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/ui_options.go:167 +0x14f
2026-02-01T19:39:22.878266486Z net/http.HandlerFunc.ServeHTTP(0xc003489045?, {0x52208b0?, 0xc0057998a8?}, 0xc000ea1410?)
2026-02-01T19:39:22.878270323Z /usr/local/go/src/net/http/server.go:2322 +0x29
2026-02-01T19:39:22.878357833Z github.com/go-openapi/runtime/middleware.Spec.func1({0x52208b0, 0xc0057998a8}, 0xc001c05e00)
2026-02-01T19:39:22.878362311Z /go/src/github.com/cilium/cilium/vendor/github.com/go-openapi/runtime/middleware/spec.go:61 +0x1a2
2026-02-01T19:39:22.878364836Z net/http.HandlerFunc.ServeHTTP(0xc000fc3440?, {0x52208b0?, 0xc0057998a8?}, 0x3?)
2026-02-01T19:39:22.878367561Z /usr/local/go/src/net/http/server.go:2322 +0x29
2026-02-01T19:39:22.878370185Z github.com/cilium/cilium/pkg/metrics.(*APIEventTSHelper).ServeHTTP(0xc000bf44b0, {0x5220a40, 0xc0029af1d0}, 0xc001c05e00)
2026-02-01T19:39:22.878372630Z /go/src/github.com/cilium/cilium/pkg/metrics/middleware.go:63 +0x1c3
2026-02-01T19:39:22.878375145Z github.com/cilium/cilium/pkg/api.(*APIPanicHandler).ServeHTTP(0xc002a62840?, {0x5220a40?, 0xc0029af1d0?}, 0x90f756?)
2026-02-01T19:39:22.878377740Z /go/src/github.com/cilium/cilium/pkg/api/apipanic.go:56 +0x62
2026-02-01T19:39:22.878380114Z net/http.(*ServeMux).ServeHTTP(0x47e239?, {0x5220a40, 0xc0029af1d0}, 0xc001c05e00)
2026-02-01T19:39:22.878382368Z /usr/local/go/src/net/http/server.go:2861 +0x1c7
2026-02-01T19:39:22.878388169Z net/http.serverHandler.ServeHTTP({0xc00172f000?}, {0x5220a40?, 0xc0029af1d0?}, 0x1?)
2026-02-01T19:39:22.878390684Z /usr/local/go/src/net/http/server.go:3340 +0x8e
2026-02-01T19:39:22.878393008Z net/http.(*conn).serve(0xc000ded0e0, {0x5232cd8, 0xc000b01890})
2026-02-01T19:39:22.878395362Z /usr/local/go/src/net/http/server.go:2109 +0x665
2026-02-01T19:39:22.878413085Z created by net/http.(*Server).Serve in goroutine 659
2026-02-01T19:39:22.878415910Z /usr/local/go/src/net/http/server.go:3493 +0x485
2026-02-01T19:39:22.878418105Z
...
2026-02-01T19:39:22.886267855Z goroutine 457 [runnable]:
2026-02-01T19:39:22.886269939Z type:.eq.log/slog.Source(0xc0042ebe60?, 0x86b0260?)
2026-02-01T19:39:22.886299654Z <autogenerated>:1 +0x7e
2026-02-01T19:39:22.886302249Z log/slog.(*Source).isEmpty(...)
2026-02-01T19:39:22.886310354Z /usr/local/go/src/log/slog/record.go:215
2026-02-01T19:39:22.886312839Z log/slog.(*handleState).appendAttr(0xc0034031d8, {{0x4afd576, 0x6}, {{}, 0x0, {0x40e8320, 0xc0042ebe60}}})
2026-02-01T19:39:22.886315223Z /usr/local/go/src/log/slog/handler.go:495 +0x39d
2026-02-01T19:39:22.886317567Z log/slog.(*commonHandler).handle(_, {{0xc25808dab42330a6, 0x11db4e1e47, 0x815bda0}, {0x4b1f8c5, 0xe}, 0xfffffffffffffffc, 0x3178d7b, {{{0x4af9c0a, 0x4}, ...}, ...}, ...})
2026-02-01T19:39:22.886319761Z /usr/local/go/src/log/slog/handler.go:306 +0x4b1
2026-02-01T19:39:22.886321996Z log/slog.(*TextHandler).Handle(_, {_, _}, {{0xc25808dab42330a6, 0x11db4e1e47, 0x815bda0}, {0x4b1f8c5, 0xe}, 0xfffffffffffffffc, 0x3178d7b, ...})
2026-02-01T19:39:22.886324520Z /usr/local/go/src/log/slog/text_handler.go:93 +0x3f
2026-02-01T19:39:22.886326754Z github.com/cilium/cilium/pkg/logging.(*multiSlogHandler).Handle(_, {_, _}, {{0xc25808dab42330a6, 0x11db4e1e47, 0x815bda0}, {0x4b1f8c5, 0xe}, 0xfffffffffffffffc, 0x3178d7b, ...})
2026-02-01T19:39:22.886329149Z /go/src/github.com/cilium/cilium/pkg/logging/multihandler.go:48 +0x198
2026-02-01T19:39:22.886331473Z log/slog.(*Logger).log(0xc000a6e0f0, {0x5232ae8?, 0x86a99c0?}, 0xfffffffffffffffc, {0x4b1f8c5, 0xe}, {0xc003403aa0, 0x2, 0x2})
2026-02-01T19:39:22.886333607Z /usr/local/go/src/log/slog/logger.go:256 +0x208
2026-02-01T19:39:22.886335731Z log/slog.(*Logger).Debug(...)
2026-02-01T19:39:22.886338015Z /usr/local/go/src/log/slog/logger.go:199
2026-02-01T19:39:22.886340199Z github.com/cilium/cilium/pkg/ipam.(*cidrPool).releaseExcessCIDRsMultiPool(0xc00108e9c0, 0x0)
2026-02-01T19:39:22.886342414Z /go/src/github.com/cilium/cilium/pkg/ipam/pool.go:206 +0x346
2026-02-01T19:39:22.886344587Z github.com/cilium/cilium/pkg/ipam.(*multiPoolManager).updateLocalNode(0xc002851220, {0x5232cd8, 0xc000a23f20})
2026-02-01T19:39:22.886347002Z /go/src/github.com/cilium/cilium/pkg/ipam/multipool_manager.go:554 +0x585
2026-02-01T19:39:22.886349376Z github.com/cilium/hive/job.(*jobTimer).start(0xc0024eaa80, {0x5232cd8, 0xc000a23f20}, 0x51c5bf0?, {0x52487e0, 0xc001c92780}, {{{0xc001a8f9c0, 0x1, 0x1}}, 0xc000a6e0f0, ...})
2026-02-01T19:39:22.886351721Z /go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/timer.go:191 +0x473
2026-02-01T19:39:22.886353805Z created by github.com/cilium/hive/job.(*group).add.func1 in goroutine 1
2026-02-01T19:39:22.886356079Z /go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/job.go:263 +0x16f
Incorrect map access happens in pkg/ipam/multipool.go, line 120:
func (c *multiPoolAllocator) Capacity() uint64 {
var capacity uint64
for _, pool := range c.manager.pools { // <<< manager.pools is accessed here without taking manager mutex
var p *cidrPool
The problem is that c.manager.pools is iterated over without holding the c.manager.mutex. All other access to pools seems to be in pkg/ipam/multipool_manager.go with proper locking.
Another goroutine is deleting the map in pkg/ipam/multipool_manager.go line 564 while executing multiPoolManager.updateLocalNode().