Skip to content

Control/Data Plane Separation #2414

@terrykong

Description

@terrykong

NeMo RL has used the driver to deal with data and control. This has led to the driver being a bottleneck for data movement since all data funnels through one rank. This is no longer tenable at the scales we are now training at.

This issue tracks the separation of data plane from the driver.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions