Fix pidfd leak in UnshareAfterEnterUserns #12167

jfernandez · 2025-08-05T04:06:03Z

UnshareAfterEnterUserns() creates a pidfd via os.StartProcess() with CLONE_PIDFD but fails to close the file descriptor in any code path, resulting in a file descriptor leak for every container that uses user namespace isolation.

The leak occurs because:

The pidfd is created when PidFD field is set in SysProcAttr
The defer block only calls PidfdSendSignal() and pidfdWaitid()
No code path calls unix.Close(pidfd) to release the file descriptor

This causes one pidfd leak per container launch when user namespace isolation is enabled (e.g., Kubernetes pods with hostUsers: false). In production environments with high container churn, this can exhaust the system's file descriptor limit.

Fix the leak by adding a defer statement immediately after process creation that ensures unix.Close(pidfd) is always called, regardless of which code path is taken. This guarantees cleanup even if the function returns early due to errors or lack of pidfd support.

This follows the same cleanup pattern already established in core/mount/mount_idmapped_utils_linux.go:getUsernsFD(), which properly closes its pidfd.

Closes: #12166
Fixes: #10607

k8s-ci-robot · 2025-08-05T04:06:13Z

Hi @jfernandez. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jfernandez · 2025-08-05T12:20:56Z

We tracked this down after noticing that open file descriptors by containerd (via the process_open_fds metric) always incremented and were only reset during containerd restarts. We wrote a bpftrace script to show all clone syscalls from containerd processes that use the CLONE_PIDFD flag.

tracepoint:raw_syscalls:sys_enter
  /comm == "containerd" && args->id == 56/  // clone syscall
  {
      if (args->args[0] & 0x1000) {  // CLONE_PIDFD flag
          @clone_tid[tid] = 1;
          @clone_flags[tid] = args->args[0];
      }
  }

tracepoint:raw_syscalls:sys_exit
/comm == "containerd" && args->id == 56 && @clone_tid[tid]/
  {
      if (args->ret > 0) {
          time("%H:%M:%S ");
          printf("| CLONE_PIDFD | child_pid=%d flags=0x%x\n", args->ret, @clone_flags[tid]);
          printf("         | STACK | \n");
          print(ustack);
          printf("\n");
      }
      delete(@clone_tid[tid]);
      delete(@clone_flags[tid]);
  }

  END
  {
      clear(@clone_tid);
      clear(@clone_flags);
  }

20:26:42 | CLONE_PIDFD | child_pid=105077 flags=0x10001011
         | STACK | 
        syscall.rawVforkSyscall.abi0+45
        syscall.forkAndExecInChild+85
        syscall.forkExec+844
        os.startProcess+754
        os.StartProcess+84
        github.com/containerd/containerd/v2/pkg/sys.UnshareAfterEnterUserns+520
        github.com/containerd/containerd/v2/internal/cri/server.(*criService).setupNetnsWithinUserns+649
        github.com/containerd/containerd/v2/internal/cri/server.(*criService).RunPodSandbox+10220
        github.com/containerd/containerd/v2/plugins/cri.(*criGRPCServer).RunPodSandbox+49
        github.com/containerd/containerd/v2/internal/cri/instrument.(*instrumentedService).RunPodSandbox+451
        k8s.io/cri-api/pkg/apis/runtime/v1._RuntimeService_RunPodSandbox_Handler.func1+206
        github.com/containerd/containerd/v2/cmd/containerd/server.New.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func11+650
        google.golang.org/grpc.getChainUnaryHandler.func1+178
        github.com/containerd/containerd/v2/cmd/containerd/server.unaryNamespaceInterceptor+101
        google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1+133
        k8s.io/cri-api/pkg/apis/runtime/v1._RuntimeService_RunPodSandbox_Handler+323
        google.golang.org/grpc.(*Server).processUnaryRPC+3627
        google.golang.org/grpc.(*Server).handleStream+3723
        google.golang.org/grpc.(*Server).serveStreams.func2.1+127
        runtime.goexit.abi0+1

austinvazquez · 2025-08-05T12:22:59Z

/ok-to-test

jfernandez · 2025-08-05T12:31:29Z

/retest

pkg/sys/unshare_linux.go

UnshareAfterEnterUserns() creates a pidfd via os.StartProcess() with CLONE_PIDFD but fails to close the file descriptor in any code path, resulting in a file descriptor leak for every container that uses user namespace isolation. The leak occurs because: - The pidfd is created when PidFD field is set in SysProcAttr - The original defer block only calls PidfdSendSignal() and pidfdWaitid() - No code path calls unix.Close(pidfd) to release the file descriptor This causes one pidfd leak per container launch when user namespace isolation is enabled (e.g., Kubernetes pods with hostUsers: false). In production environments with high container churn, this can exhaust the system's file descriptor limit. Fix the leak by adding a defer statement immediately after process creation that ensures unix.Close(pidfd) is always called, regardless of which code path is taken. This guarantees cleanup even if the function returns early due to errors or lack of pidfd support. This follows the same cleanup pattern already established in core/mount/mount_idmapped_utils_linux.go:getUsernsFD() which properly closes its pidfd. Closes: containerd#12166 Signed-off-by: Jose Fernandez <josef@netflix.com>

fuweid · 2025-08-07T03:10:30Z

Hi @jfernandez thanks for the patch. The change looks good.

Just have quick question: did you see failed to ensure the kernel supports pidfd in the log?

fuweid

LGTM

jfernandez · 2025-08-07T03:44:07Z

Just have quick question: did you see failed to ensure the kernel supports pidfd in the log?

We do not see that error message in our logs. We also confirmed that our kernel supports pidfd.

jfernandez · 2025-08-07T03:56:54Z

/retest

dcantah

Thanks! LGTM

rata · 2025-08-07T12:48:44Z

Shall we cherry-pick this to 2.1 and 2.0?

fuweid · 2025-08-07T13:44:22Z

/cherry-pick release/2.0
/cherry-pick release/2.1

k8s-infra-cherrypick-robot · 2025-08-07T13:45:08Z

@fuweid: new pull request created: #12178

Details

In response to this:

/cherry-pick release/2.0
/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2025-08-07T13:45:44Z

@fuweid: new pull request created: #12179

Details

In response to this:

/cherry-pick release/2.0
/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

github-project-automation bot moved this to Needs Triage in Pull Request Review Aug 5, 2025

github-project-automation bot added this to Pull Request Review Aug 5, 2025

k8s-ci-robot added needs-ok-to-test size/XS labels Aug 5, 2025

dosubot bot added area/runtime Runtime kind/bug labels Aug 5, 2025

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Aug 5, 2025

tych0 reviewed Aug 5, 2025

View reviewed changes

pkg/sys/unshare_linux.go Show resolved Hide resolved

dcantah reviewed Aug 6, 2025

View reviewed changes

pkg/sys/unshare_linux.go Show resolved Hide resolved

jfernandez force-pushed the pidfd-leak branch from b6225fa to 45e02e1 Compare August 7, 2025 02:37

fuweid approved these changes Aug 7, 2025

View reviewed changes

fuweid added cherry-pick/2.0.x Change to be cherry picked to release/2.0 branch cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch impact/changelog labels Aug 7, 2025

fuweid enabled auto-merge August 7, 2025 03:38

dcantah approved these changes Aug 7, 2025

View reviewed changes

fuweid added this pull request to the merge queue Aug 7, 2025

github-project-automation bot moved this from Needs Triage to Review In Progress in Pull Request Review Aug 7, 2025

Merged via the queue into containerd:main with commit ef68642 Aug 7, 2025
120 of 122 checks passed

github-project-automation bot moved this from Review In Progress to Done in Pull Request Review Aug 7, 2025

k8s-infra-cherrypick-robot mentioned this pull request Aug 7, 2025

[release/2.0] Fix pidfd leak in UnshareAfterEnterUserns #12178

Merged

k8s-infra-cherrypick-robot mentioned this pull request Aug 7, 2025

[release/2.1] sys: fix pidfd leak in UnshareAfterEnterUserns #12179

Merged

dmcgowan changed the title ~~sys: fix pidfd leak in UnshareAfterEnterUserns~~ Fix pidfd leak in UnshareAfterEnterUserns Sep 11, 2025

dmcgowan mentioned this pull request Sep 11, 2025

Prepare release notes for v2.2.0-beta.0 #12293

Merged

dmcgowan mentioned this pull request Oct 26, 2025

Prepare release notes for v2.2.0-rc.0 #12413

Merged

dmcgowan mentioned this pull request Nov 5, 2025

Prepare release notes for v2.2.0 #12473

Merged

Fix pidfd leak in UnshareAfterEnterUserns #12167

Fix pidfd leak in UnshareAfterEnterUserns #12167

Uh oh!

Conversation

jfernandez commented Aug 5, 2025 • edited by fuweid Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Aug 5, 2025

Uh oh!

jfernandez commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

austinvazquez commented Aug 5, 2025

Uh oh!

jfernandez commented Aug 5, 2025

Uh oh!

Uh oh!

Uh oh!

fuweid commented Aug 7, 2025

Uh oh!

fuweid left a comment

Choose a reason for hiding this comment

Uh oh!

jfernandez commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfernandez commented Aug 7, 2025

Uh oh!

dcantah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rata commented Aug 7, 2025

Uh oh!

fuweid commented Aug 7, 2025

Uh oh!

k8s-infra-cherrypick-robot commented Aug 7, 2025

Uh oh!

k8s-infra-cherrypick-robot commented Aug 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jfernandez commented Aug 5, 2025 •

edited by fuweid

Loading

jfernandez commented Aug 5, 2025 •

edited

Loading

jfernandez commented Aug 7, 2025 •

edited

Loading