Skip to content

Add NVIDIA APIs#4182

Merged
arnaldo2792 merged 4 commits intobottlerocket-os:developfrom
arnaldo2792:nvidia-api-kit
Sep 12, 2024
Merged

Add NVIDIA APIs#4182
arnaldo2792 merged 4 commits intobottlerocket-os:developfrom
arnaldo2792:nvidia-api-kit

Conversation

@arnaldo2792
Copy link
Copy Markdown
Contributor

@arnaldo2792 arnaldo2792 commented Sep 9, 2024 •
edited
Loading

Issue number:

N / A

Description of changes:

This includes two new APIs to all the NVIDIA variants:

  • settings.nvidia-container-runtime: used to configure the NVIDIA container toolkit behavior
  • settings.kubelet-device-plugin.nvidia: used to expose configurations for the NVIDIA k8s device plugin

Testing done:

As part of bottlerocket-os/bottlerocket-settings-sdk#60 and bottlerocket-os/bottlerocket-core-kit#132

  1. Instance joined the cluster:
NAME                                           STATUS   ROLES    AGE    VERSION
ip-192-168-41-204.us-west-2.compute.internal   Ready    <none>   22s    v1.30.1-eks-e564799
  1. The safe defaults were used, which prevented the containers from accessing all the GPUs:
└─> ❯ k exec safe-defaults-d7g2j -- env | rg NVIDIA_VISIBLE_DEVICES
NVIDIA_VISIBLE_DEVICES=all
└─> ❯ k exec safe-defaults-d7g2j -- cat /proc/self/mountinfo | rg nvidia
┌───────────────────> ~/Code/work/by-feature/nvidia-settings-api/testing on Fedora
└─> ❯
  1. Files were generated using the new APIs
bash-5.1# cat /etc/systemd/system/nvidia-k8s-device-plugin.service.d/exec-start.conf
[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-device-plugin --config-file=/etc/nvidia-k8s-device-plugin/settings.yaml
bash-5.1# cat /etc/nvidia-container-runtime/config.toml
### generated from the template file ###
accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

[nvidia-container-cli]
root = "/"
path = "/usr/bin/nvidia-container-cli"
environment = []
ldconfig = "@/sbin/ldconfig"
bash-5.1#
  1. Device Plugin was restarted after the values changed to allow containers access all GPUs
bash-5.1# apiclient get settings.kubelet-device-plugin
{
  "settings": {
    "kubelet-device-plugin": {
      "nvidia": {
        "device-id-strategy": "index",
        "device-list-strategy": "volume-mounts",
        "pass-device-specs": true
      }
    }
  }
}
bash-5.1# apiclient set kubelet-device-plugin.nvidia.device-list-strategy=envvar
bash-5.1# systemctl status nvidia-k8s-device-plugin.service
● nvidia-k8s-device-plugin.service - Start NVIDIA kubernetes device plugin
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/nvidia-k8s-device-plugin.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/nvidia-k8s-device-plugin.service.d
             └─exec-start.conf
     Active: active (running) since Mon 2024-09-09 15:28:10 UTC; 9s ago
   Main PID: 11094 (nvidia-device-p)
      Tasks: 9 (limit: 38028)
     Memory: 17.9M
        CPU: 9.102s
     CGroup: /system.slice/nvidia-k8s-device-plugin.service
             └─11094 /usr/bin/nvidia-device-plugin --config-file=/etc/nvidia-k8s-device-plugin/settings.yaml

Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]:     ]
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]:   },
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]:   "sharing": {
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]:     "timeSlicing": {}
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]:   }
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]: }
Sep 09 15:28:10 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]: I0909 15:28:10.178566   11094 main.go:317] Retrieving plugins.
Sep 09 15:28:17 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]: I0909 15:28:17.416865   11094 server.go:216] Starting GRPC server for 'nvidia.com/gpu'
Sep 09 15:28:17 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]: I0909 15:28:17.417380   11094 server.go:147] Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
Sep 09 15:28:17 ip-192-168-41-204.us-west-2.compute.internal nvidia-device-plugin[11094]: I0909 15:28:17.419653   11094 server.go:154] Registered device plugin for 'nvidia.com/gpu' with Kubelet
bash-5.1#
  1. Newly created container has access to all the GPUs without requesting any, after the configurations were updated:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: safe-defaults
spec:
  selector:
    matchLabels:
      name: safe-defaults
  template:
    metadata:
      labels:
        name: safe-defaults
    spec:
      # No GPUs requested
      containers:
        - name: safe-defaults
          image: nvidia/cuda:12.4.1-cudnn-devel-rockylinux8
          command: ['sh', '-c', 'sleep infinity']
└─> ❯ k exec safe-defaults-cnsqn -- nvidia-smi
Mon Sep  9 15:31:38 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.4     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A10G                    Off | 00000000:00:1E.0 Off |                    0 |
|  0%   25C    P8               8W / 300W |      0MiB / 23028MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
┌───────────────────> ~/Code/work/by-feature/nvidia-settings-api/testing on Fedora
└─> ❯
  1. In k8s 1.30, 1.29, 1.28, 1.27, I performed upgrade/downgrade testing. Confirmed that the new APIs were available in the upgrade, and the values were removed on a downgrade. I also performed the same testing described above, to make sure the defaults are set and that the containers with NVIDIA_VISIBLE_DEVICES=all can access all GPUs when the configurations to allow this are set.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

piyush-jena
piyush-jena reviewed Sep 10, 2024
View reviewed changes
@arnaldo2792 arnaldo2792 force-pushed the nvidia-api-kit branch 2 times, most recently from cc357d4 to d8b052f Compare September 10, 2024 23:12
@arnaldo2792
Copy link
Copy Markdown
Contributor Author

arnaldo2792 commented Sep 10, 2024

(forced push includes rebase)

@arnaldo2792
Copy link
Copy Markdown
Contributor Author

arnaldo2792 commented Sep 11, 2024

Forced push includes:

  • Use the workspace migration helpers
  • Add bootstrap commands API to NVIDIA settings plugins
  • Minor stylistic edits to migrations

@arnaldo2792 arnaldo2792 mentioned this pull request Sep 12, 2024
Closed
@arnaldo2792 arnaldo2792 marked this pull request as ready for review September 12, 2024 17:02
@arnaldo2792
Copy link
Copy Markdown
Contributor Author

arnaldo2792 commented Sep 12, 2024

(rebased develop and drop hack commit)

@arnaldo2792
settings-plugins: add aws-k8s-nvidia plugin
a157d54 
Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>
@arnaldo2792
settings-defaults: add defaults for new NVIDIA APIs
2a4d3ab 
Add settings defaults for nvidia-container-runtime and kubelet-device-plugin

Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>
@arnaldo2792
migration: add migrations for new NVIDIA APIs
2b09476 
Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>
@arnaldo2792
sources: bump Settings SDK version
d3d8b35 
Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>
@arnaldo2792
Copy link
Copy Markdown
Contributor Author

arnaldo2792 commented Sep 12, 2024

Forced push fixes the settings plugins definitions for aws-k8s-1.31

ginglis13
ginglis13 approved these changes Sep 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@ginglis13 ginglis13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just some small nits

sources/settings-plugins/aws-k8s-nvidia/Cargo.toml
@@ -0,0 +1,19 @@
[package]
name = "settings-plugin-aws-k8s-nvidia"
version = "0.1.0"
Copy link
Copy Markdown
Contributor

@ginglis13 ginglis13 Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add authors here?

sources/settings-migrations/v1.23.0/nvidia-container-runtime-settings/Cargo.toml
[package]
name = "nvidia-container-runtime-settings"
version = "0.1.0"
authors = ["Monirul Islam <monirulu@amazon.com>"]
Copy link
Copy Markdown
Contributor

@ginglis13 ginglis13 Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slight nit, should @arnaldo2792 be a listed author as well? (and for other crates added in this PR)

Copy link
Copy Markdown
Contributor Author

@arnaldo2792 arnaldo2792 Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to keep the ownership of the work since the commits will have my name. My changes to that code were minimal 👍

piyush-jena
piyush-jena approved these changes Sep 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@piyush-jena piyush-jena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

yeazelm
yeazelm approved these changes Sep 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@yeazelm yeazelm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@arnaldo2792 arnaldo2792 merged commit 26ec2cc into bottlerocket-os:develop Sep 12, 2024
@yeazelm yeazelm mentioned this pull request Sep 12, 2024
Closed
3 tasks
@arnaldo2792 arnaldo2792 mentioned this pull request Sep 18, 2024
Closed
@KCSesh KCSesh mentioned this pull request Oct 8, 2024
Merged
@vigh-m vigh-m mentioned this pull request Dec 24, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@piyush-jena piyush-jena piyush-jena approved these changes

@ginglis13 ginglis13 ginglis13 approved these changes

+1 more reviewer

@yeazelm yeazelm yeazelm approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@arnaldo2792 @piyush-jena @ginglis13 @yeazelm