Skip to content

fix(datapath): disable VF info collection in netlink#43517

Merged
ti-mo merged 1 commit intocilium:mainfrom
nebius:IK8SNET-310
Feb 5, 2026
Merged

fix(datapath): disable VF info collection in netlink#43517
ti-mo merged 1 commit intocilium:mainfrom
nebius:IK8SNET-310

Conversation

@pasteley
Copy link
Copy Markdown
Contributor

@pasteley pasteley commented Dec 28, 2025

Disable Virtual Function information collection in netlink handle operations to avoid hold of rtnl_mutex on systems with SR-IOV devices.

Fixes: #43516

Reduces rtnl_mutex contention on SR-IOV nodes by not requesting VF information in netlink RTM_GETLINK operations

@pasteley pasteley requested review from a team as code owners December 28, 2025 23:44
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Dec 28, 2025
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Dec 28, 2025
@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 6, 2026

overall code looks good to me. there's a failed test, but it could be a flake. havent looked into it yet. hopefully someone can hit the retry
other than that, you're missing the release note in the pr description. mind adding it? 🙏🏽

@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 6, 2026

/ci-integration

@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 6, 2026

/test

@ti-mo
Copy link
Copy Markdown
Contributor

ti-mo commented Jan 6, 2026

Hi, what problem does this solve, exactly? Is there an existing issue that points out why the lock being held is a problem? Thanks.

@pasteley
Copy link
Copy Markdown
Contributor Author

pasteley commented Jan 6, 2026

Is there an existing issue that points out why the lock being held is a problem? Thanks.

@ti-mo yes, you can find it mentioned in PR description: #43516

@ti-mo ti-mo added the release-note/misc This PR makes changes that have no direct user impact. label Jan 6, 2026
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jan 6, 2026
@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 7, 2026

@pasteley , can you add a release note message to the pr description?
something similar to this one (in the code block): #42896 (comment)

@joestringer joestringer added release-note/bug This PR fixes an issue in a previous release of Cilium. needs-backport/1.18 This PR / issue needs backporting to the v1.18 branch and removed release-note/misc This PR makes changes that have no direct user impact. labels Jan 9, 2026
@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 9, 2026

/test

@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 9, 2026

/ci-ginkgo

@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 9, 2026

@bersoare
Copy link
Copy Markdown
Contributor

bersoare commented Jan 9, 2026

/ci-l7

Copy link
Copy Markdown
Contributor

@ti-mo ti-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, not sure how I missed the issue link. Left a few comments outlining my thoughts and the way forward for this package.

@aanm aanm added the needs-backport/1.19 This PR / issue needs backporting to the v1.19 branch label Jan 14, 2026
Disable Virtual Function information collection in netlink handle
operations to avoid hold of `rtnl_mutex` on systems with SR-IOV devices.

Fixes: cilium#43516
Signed-off-by: pasteley <ceasebeing@gmail.com>
Copy link
Copy Markdown
Contributor

@ti-mo ti-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, thanks! 💯

@ti-mo
Copy link
Copy Markdown
Contributor

ti-mo commented Jan 27, 2026

/test

@ti-mo ti-mo enabled auto-merge January 27, 2026 13:21
Copy link
Copy Markdown
Contributor

@ti-mo ti-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized this completely sidesteps the purpose of the safenetlink package, since NewHandle returns a netlink.Handle, whose methods aren't intercepted by the safenetlink package. I think we need to introduce a safenetlink.Handle that wraps the netlink one and also wraps its methods with a retry loop.

@gandro PTAL

@gandro
Copy link
Copy Markdown
Member

gandro commented Feb 4, 2026

Took a look - since we moved to forbidigo, we will actually detect calls to the netlink.Handle. So the PR here doesn't sidestep our linter, it will still complain if you call unsafe functions on the handle returned by safenetlink.NewHandle.

Calling unsafe functions on the handle require you to use safenetlink.WithRetry* - here's an example from our codebase:
https://github.com/nebius/cilium/blob/d06fe451b641338b41e4542661c1eb34fda4bd85/pkg/datapath/linux/route/reconciler/reconciler.go#L349-L356

If we introduce a wrapper, then we have to make sure that all wrapped functions are safe. So from a safety perspective, I think this PR is fine as is. But it would probably be a potential improvement if we wrapped netlink.Handle to offer safe methods as well.

@gandro
Copy link
Copy Markdown
Member

gandro commented Feb 4, 2026

On a only slightly related node: What we could consider is making sure that netlink.NewHandle is also forbidden by adding it to the list here:

- pattern: ^netlink\.(Handle\.)?("AddrList|BridgeVlanList|ChainList|ClassList|ConntrackTableList|DevLinkGetDeviceList|DevLinkGetAllPortList|DevlinkGetDeviceParams|FilterList|FouList|GenlFamilyList|GTPPDPList|LinkByName|LinkByAlias|LinkList|LinkSubscribeWithOptions|NeighList|NeighProxyList|NeighListExecute|LinkGetProtinfo|QdiscList|RdmaLinkList|RdmaLinkByName|RdmaLinkDel|RouteList|RouteListFiltered|RouteListFilteredIter|RouteSubscribeWithOptions|RuleList|RuleListFiltered|SocketGet|SocketDiagTCPInfo|SocketDiagTCP|SocketDiagUDPInfo|SocketDiagUDP|UnixSocketDiagInfo|UnixSocketDiag|SocketXDPGetInfo|SocketDiagXDP|VDPAGetDevList|VDPAGetDevConfigList|VDPAGetMGMTDevList|XfrmPolicyList|XfrmStateList)

That way, new code would have to always use safenetlink.NewHandle (or explicitly opt out if they want to see VFs)

@ti-mo
Copy link
Copy Markdown
Contributor

ti-mo commented Feb 4, 2026

@pasteley I'd like to hold off on this PR a bit as it runs the risk of regressing. In its current state, it doesn't handle netlink.ErrDumpInterrupted. I've created an issue in the library to hopefully be able to remove safenetlink in the medium term: vishvananda/netlink#1163.

@gandro
Copy link
Copy Markdown
Member

gandro commented Feb 4, 2026

@pasteley I'd like to hold off on this PR a bit as it runs the risk of regressing. In its current state, it doesn't handle netlink.ErrDumpInterrupted. I've created an issue in the library to hopefully be able to remove safenetlink in the medium term: vishvananda/netlink#1163.

I'm not sure I follow? Our linter does complain if you call a function on netlink.Handle that may return netlink.ErrDumpInterrupted, even if the handle was constructed via safenetlink.NewHandle introduced by this PR.

@ti-mo
Copy link
Copy Markdown
Contributor

ti-mo commented Feb 4, 2026

I'm not sure I follow? Our linter does complain if you call a function on netlink.Handle that may return netlink.ErrDumpInterrupted, even if the handle was constructed via safenetlink.NewHandle introduced by this PR.

Ohhh, I see. We already have a bunch of places where Handle is used, forbidigo silenced, and an explicit retry loop added. Of course, then we can merge this as-is. Just need to figure out how to disable VF collection by default if/when we manage to get a retry mechanism added upstream. I'll spend some time on that when I make an upstream proposal.

@pasteley
Copy link
Copy Markdown
Contributor Author

pasteley commented Feb 4, 2026

@cilium/sig-datapath ptal

Copy link
Copy Markdown
Contributor

@smagnani96 smagnani96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I don't see us using the retrieved link.Attrs().Vfs in our codebase nowadays.

@ti-mo ti-mo added this pull request to the merge queue Feb 5, 2026
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Feb 5, 2026
Merged via the queue into cilium:main with commit cc2d27c Feb 5, 2026
78 checks passed
@Artyop Artyop mentioned this pull request Feb 10, 2026
6 tasks
@Artyop Artyop added backport-pending/1.19 The backport for Cilium 1.19.x for this PR is in progress. and removed needs-backport/1.19 This PR / issue needs backporting to the v1.19 branch labels Feb 10, 2026
@Artyop Artyop mentioned this pull request Feb 10, 2026
2 tasks
@Artyop Artyop added backport-pending/1.18 The backport for Cilium 1.18.x for this PR is in progress. and removed needs-backport/1.18 This PR / issue needs backporting to the v1.18 branch labels Feb 10, 2026
@github-actions github-actions bot added backport-done/1.18 The backport for Cilium 1.18.x for this PR is done. backport-done/1.19 The backport for Cilium 1.19.x for this PR is done. and removed backport-pending/1.18 The backport for Cilium 1.18.x for this PR is in progress. backport-pending/1.19 The backport for Cilium 1.19.x for this PR is in progress. labels Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-done/1.18 The backport for Cilium 1.18.x for this PR is done. backport-done/1.19 The backport for Cilium 1.19.x for this PR is done. kind/community-contribution This was a contribution made by a community member. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cilium can hold rtnl_mutex for long on SR-IOV hosts due to VF info filling

9 participants