implement NRI plugin server to inject management CDI devices by tariq1890 · Pull Request #1498 · NVIDIA/nvidia-container-toolkit

tariq1890 · 2025-11-22T00:57:56Z

This PR adds support for an NRI Plugin which will intercept CreateContainer events to inject CDI devices to management containers.

Here is a summary of the changes:

NRI Plugin will listen for CreateContainer events, check if the reserved annotation (for e.g. - "nvidia.cdi.k8s.io/container.{{container-name}}" ) exists in the pod
The NRI Plugin is disabled by default.
When enabled, the runtime.Setup method is skipped. Runtime config TOML will not be modified and the runtime will no longer need to be restarted
If the user is using a container runtime version (for e.g. containerd v1.7.30) where NRI and CDI are disabled by default, they will be have to be enabled out of a band as a pre-requisite

Future Scope

Should we add a flag to force runtime class creation when NRI plugin is enabled?
Add mechanisms to restrict who/which pods can inject these devices. One idea is to set a list of allowed namespaces. The NRI Plugin could limit CDI device injections to pods in an allowed namespace.

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

elezar

Thanks @tariq1890.

As discussed yesterday, I think the approach is good in general. It would be good to define what level of control we want to afford users w.r.t configuring runtimes. (e.g. Do we want to allow them to enable NRI (and CDI) and configure the runtime classes?).

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

elezar · 2026-01-13T12:44:09Z

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

+	// nriCDIDeviceKey is the prefix of the key used for CDI device annotations
+	nriCDIDeviceKey = nriCDIDeviceDomain + "/container"
+	// defaultNRISocket represents the default path of the NRI socket
+	defaultNRISocket = "/var/run/nri/nri.sock"


Let's not define this here and instead ensure that we pass in the default value. (We probably want to add the socket as a CLI flag so that we COULD override it.

Nit: We also have api.DefaultSocketPath that we could use.

elezar · 2026-01-13T12:45:40Z

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

+	return cdiDevices, nil
+}
+
+func getCDIDeviceFromAnnotation(annotations map[string]string, ctr string) string {


nit: Technically we're just getting the annotation value here -- which MAY be a comma-separated list of devices. Do we want to rename the function?

Does getCDIDeviceNamesFromAnnotation work? Open to suggestions here

I have changed the method names, please check now

I've renamed the function

elezar · 2026-01-13T12:53:03Z

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

+	nriPluginAnnotationKey := fmt.Sprintf("%s.%s", nriCDIDeviceKey, ctr)
+	if value, ok := annotations[nriPluginAnnotationKey]; ok {
+		return value
+	}
+
+	// If there isn't an exact match, we look for a wildcard character match
+	for annotationKey := range annotations {
+		if after, found := strings.CutPrefix(annotationKey, fmt.Sprintf("%s.", nriCDIDeviceKey)); found {
+			if after == "*" {
+				return annotations[annotationKey]
+			}
+		}
+	}
+
+	return ""


We could introduce a new constant that is equal to "nvidia.cdi.k8s.io/container." to simplify this quite a bit. Also note that we need not use strings.CutPrefix, since we known the full key for the "wildcard" annotation beforehand. (It's also possible to use a recursive implementation, but this is not required).

Suggested change

nriPluginAnnotationKey := fmt.Sprintf("%s.%s", nriCDIDeviceKey, ctr)

if value, ok := annotations[nriPluginAnnotationKey]; ok {

return value

}

// If there isn't an exact match, we look for a wildcard character match

for annotationKey := range annotations {

if after, found := strings.CutPrefix(annotationKey, fmt.Sprintf("%s.", nriCDIDeviceKey)); found {

if after == "*" {

return annotations[annotationKey]

}

}

}

return ""

nriPluginAnnotationKeyPrefix := nriCDIDeviceKey + "."

containerNriPluginAnnotationKey := nriPluginAnnotationKeyPrefix + ctr

if value, ok := annotations[containerNriPluginAnnotationKey]; ok {

return value

}

if ctr == "*" {

return ""

}

// If there isn't an exact match, we look for the key that matches ALL containers, indicated by the container name '*':

return getDeviceFromAnnoations(annotations, "*")

We could introduce a new constant that is equal to "nvidia.cdi.k8s.io/container." to simplify this quite a bit. Also note that we need not use strings.CutPrefix

Agreed. I have made the changes to directly lookup the necessary keys instead of looping through the entire keyset of the map. Thanks for the suggestion!

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

cmd/nvidia-ctk-installer/main.go

elezar · 2026-01-13T13:24:12Z

cmd/nvidia-ctk-installer/container/runtime/runtime.go

 	// defaultRuntimeName specifies the NVIDIA runtime to be use as the default runtime if setting the default runtime is enabled
 	defaultRuntimeName   = "nvidia"
 	defaultHostRootMount = "/host"
+	defaultNRIPluginIdx  = "10"


Question: why 10?

I based this off the device-injector implementation from the upstream nri samples

Here are the docs for the example plugin in the upstream nri repo where they instruct users to provide 10 as the value.

elezar · 2026-01-13T13:27:24Z

cmd/nvidia-ctk-installer/main.go

-	if err != nil {
-		return fmt.Errorf("unable to setup runtime: %v", err)
+	// if NRI is enabled, we don't need to modify the container runtime config TOML.
+	if !o.runtimeOptions.EnableNRI {


The behaviour of this main funtion should NOT depend on the runtimeOptions. If we wanted to do a quick return in runtime.Setup based on the option, we can do that there.

cmd/nvidia-ctk-installer/main.go

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

tariq1890 · 2026-01-14T03:16:48Z

@elezar It looks like * is not an allowed character in annotation keys

The Pod "ubuntu" is invalid: metadata.annotations: Invalid value: "nvidia.cdi.k8s.io/container.*": name part must consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9.]*)?[A-Za-z0-9]')

I am thinking of just using pod as the reserved annotation suffix to trigger the injection of the CDI device(s) to every container in the pod.

So examples of supported annotation k-v pairs are as follows:

nvidia.cdi.k8s.io/container.nvidia-dcgm-exporter: management.nvidia.com/gpu=all
nvidia.cdi.k8s.io/pod: management.nvidia.com/gpu=all

tariq1890 · 2026-01-14T04:07:01Z

@elezar Thanks for the thorough review! I have addressed your comments and the PR is ready for another round of review

klihub · 2026-01-14T08:06:26Z

I am thinking of just using pod as the reserved annotation suffix to trigger the injection of the CDI device(s) to every container in the pod.

So examples of supported annotation k-v pairs are as follows:
nvidia.cdi.k8s.io/container.nvidia-dcgm-exporter: management.nvidia.com/gpu=all
nvidia.cdi.k8s.io/pod: management.nvidia.com/gpu=all

@tariq1890 👍 We have a similar convention with the addition that an omitted /pod suffix in the key also implies pod scope.

tariq1890 · 2026-01-14T23:27:33Z

@klihub Even better! Thank you for this suggestion. Always happy to remove code and reuse library code instead :)

cmd/nvidia-ctk-installer/main.go

tariq1890 · 2026-01-16T07:30:02Z

cmd/nvidia-ctk-installer/main.go

-		err = runtime.Cleanup(c, &o.runtimeOptions, o.runtime)
-		if err != nil {
-			return fmt.Errorf("unable to cleanup runtime: %v", err)
+		if !o.enableNRIPlugin {


@elezar This should no longer be required once #1521 gets in. I can make the necessary changes in a follow-up if this PR gets merged before #1521

Sure let's keep these separate for now. I can rebase #1521 once we merge this.

elezar

Thanks @tariq1890 (I just realised that I had a bunch of pending comments here).

This looks good now.

elezar · 2026-01-15T10:06:37Z

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

+}
+
+func parseCDIDevices(pod *api.PodSandbox, key, container string) []string {
+	cdiDeviceNames, ok := plugin.GetEffectiveAnnotation(pod, key, container)


Ok. So we've switched to using the format as implemented in the NRI plugin directly? That's better. We may need to update our documentation / implementation to use nvidia.cdi.k8s.io/pod for the cases where all containers need access to the device.

Correct. Yes, I will make sure that this is reflected in our documentation

elezar · 2026-01-15T10:26:57Z

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

+func (p *Plugin) Start(ctx context.Context, nriSocketPath, nriPluginIdx string) error {
+	pluginOpts := []stub.Option{
+		stub.WithPluginIdx(nriPluginIdx),
+		stub.WithLogger(toNriLogger{p.logger}),


Not for this PR: Note that the stub has a WithOnClose() option that allows one to specify a callback when connection is lost. Does it make sense to at least pass a function that logs so that we can collect data on when / how often this happens?

Sure, I can look into that

This is done

elezar · 2026-01-15T10:31:20Z

cmd/nvidia-ctk-installer/container/runtime/runtime.go

+	// if NRI is enabled, we don't need to modify the container runtime config TOML.
+	if opts.EnableNRI {
+		return nil
+	}


As I have called out, I don't know whether this is something that we want to do in the long term, but I'm OK to have this be the case for the time being. (If we also get the changes from #1521 in then a user can explicitly control this -- or at least opt back in to configuring things).

Not a blocker though.

I have since moved this conditional out of the method, but agreed that we would no longer need that conditional after #1521 is merged

cmd/nvidia-ctk-installer/container/runtime/runtime.go

elezar · 2026-01-16T08:08:56Z

cmd/nvidia-ctk-installer/main.go

-		err = runtime.Cleanup(c, &o.runtimeOptions, o.runtime)
-		if err != nil {
-			return fmt.Errorf("unable to cleanup runtime: %v", err)
+		if !o.enableNRIPlugin {


Sure let's keep these separate for now. I can rebase #1521 once we merge this.

cdesiniotis

@tariq1890 thanks for the work on this! This lgtm, just left a few minor typos / nits.

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go

Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>

tariq1890 force-pushed the nri-plugin-server branch 12 times, most recently from 933aa39 to 446d2f0 Compare November 22, 2025 19:07

tariq1890 force-pushed the nri-plugin-server branch 11 times, most recently from 94a00c8 to cbd8cf9 Compare December 4, 2025 07:20

tariq1890 marked this pull request as ready for review December 4, 2025 07:44

tariq1890 requested review from cdesiniotis, elezar and klueska December 4, 2025 07:44

tariq1890 force-pushed the nri-plugin-server branch from cbd8cf9 to 96b61e9 Compare December 5, 2025 03:49

elezar reviewed Dec 5, 2025

View reviewed changes

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go Outdated Show resolved Hide resolved

elezar reviewed Dec 5, 2025

View reviewed changes

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go Outdated Show resolved Hide resolved

elezar reviewed Jan 13, 2026

View reviewed changes

tariq1890 force-pushed the nri-plugin-server branch 4 times, most recently from 954d67e to b2d93d8 Compare January 14, 2026 00:00

tariq1890 force-pushed the nri-plugin-server branch 2 times, most recently from 87ca585 to 3bb5bce Compare January 14, 2026 03:48

tariq1890 force-pushed the nri-plugin-server branch from 3bb5bce to 0a2bc02 Compare January 14, 2026 23:26

tariq1890 force-pushed the nri-plugin-server branch 2 times, most recently from d3c4df6 to 4a28997 Compare January 15, 2026 07:24

This was referenced Jan 15, 2026

elezar comments for NRI plugin server #1580

Closed

elezar comments for NRI plugin server #1581

Closed

tariq1890 force-pushed the nri-plugin-server branch 5 times, most recently from 9078cfe to f15030c Compare January 15, 2026 19:17

tariq1890 commented Jan 16, 2026

View reviewed changes

elezar approved these changes Jan 16, 2026

View reviewed changes

tariq1890 force-pushed the nri-plugin-server branch from f15030c to bcfff63 Compare January 16, 2026 19:55

cdesiniotis approved these changes Jan 16, 2026

View reviewed changes

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go Outdated Show resolved Hide resolved

cmd/nvidia-ctk-installer/container/runtime/nri/plugin.go Outdated Show resolved Hide resolved

tariq1890 force-pushed the nri-plugin-server branch from bcfff63 to e31f356 Compare January 16, 2026 21:01

implement NRI plugin server to inject management CDI devices

93cb491

Signed-off-by: Tariq Ibrahim <tibrahim@nvidia.com>

tariq1890 force-pushed the nri-plugin-server branch from e31f356 to 93cb491 Compare January 16, 2026 22:41

tariq1890 merged commit d7eb81b into main Jan 16, 2026
16 checks passed

tariq1890 deleted the nri-plugin-server branch January 16, 2026 23:14

Conversation

tariq1890 commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Future Scope

Uh oh!

Uh oh!

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tariq1890 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tariq1890 commented Jan 14, 2026

Uh oh!

klihub commented Jan 14, 2026

Uh oh!

tariq1890 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tariq1890 commented Nov 22, 2025 •

edited

Loading

tariq1890 commented Jan 14, 2026 •

edited

Loading

tariq1890 commented Jan 14, 2026 •

edited

Loading