Skip to content

Treat zero cluster UUID as non-fabric-attached#54

Merged
elezar merged 1 commit intoNVIDIA:mainfrom
elezar:fix-is-fabric-attached
Feb 12, 2025
Merged

Treat zero cluster UUID as non-fabric-attached#54
elezar merged 1 commit intoNVIDIA:mainfrom
elezar:fix-is-fabric-attached

Conversation

@elezar
Copy link
Member

@elezar elezar commented Feb 12, 2025

On systems which are NVLink-capable, but not attached to the fabric we see:

[root@nvidia-device-plugin-daemonset-w494g /]# nvidia-smi -q | grep ClusterUUID
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
[root@nvidia-device-plugin-daemonset-w494g /]# nvidia-smi -q | grep CliqueId
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0
        CliqueId                          : 0

This change treats zero ClusterUUID values as not-attached.

@elezar elezar requested review from jgehrcke and klueska February 12, 2025 13:14
@elezar elezar self-assigned this Feb 12, 2025
if ret != nvml.SUCCESS {
return false, fmt.Errorf("error getting GPU Fabric Info: %v", ret)
}
if info.ClusterUuid == [16]uint8{} || info.CliqueId == 0 {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My one question would be whether || is the correct operator here. Is a clique ID of 0 valid if the cluster UUID is set?

Copy link
Collaborator

@klueska klueska Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, clique id 0 is valid if clusterUUID is set -- in fact this is how our GH200 system has it configured

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just do the following (but possibly move it lower depending on my comment below):

if info.ClusterUuid == [16]uint8{} {

Comment on lines +222 to +224
if info.ClusterUuid == [16]uint8{} || info.CliqueId == 0 {
return false, nil
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this come after verifying that GPU_FABRIC_STATE_COMPLETED?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it probably should.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar force-pushed the fix-is-fabric-attached branch from fdb3f60 to 311c577 Compare February 12, 2025 15:29
@elezar elezar requested a review from klueska February 12, 2025 15:30
@elezar elezar changed the title Treat zero cluster UUID or clique ID as non-fabric-attached Treat zero cluster UUID as non-fabric-attached Feb 12, 2025
@elezar elezar merged commit e9b8903 into NVIDIA:main Feb 12, 2025
4 checks passed
@elezar elezar deleted the fix-is-fabric-attached branch February 12, 2025 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants