Skip to content

Add Hybrid CP arg to initialize_model_parallel#3259

Merged
yaoyu-33 merged 1 commit into
mainfrom
pmannan/hybrid_cp_mlm_init
Apr 10, 2026
Merged

Add Hybrid CP arg to initialize_model_parallel#3259
yaoyu-33 merged 1 commit into
mainfrom
pmannan/hybrid_cp_mlm_init

Conversation

@parthmannan

@parthmannan parthmannan commented Apr 10, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Adds support to enable Hybrid CP

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Changelog

  • Add specific line by line info of high level changes in this PR.

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • Related to # (issue)

Summary by CodeRabbit

  • New Features
    • Added support for hybrid context parallel configuration during distributed training initialization.

@parthmannan
Add Hybrid CP arg to initialize_model_parallel to ensure NCCL group s…
7c739ef 
…etup

Signed-off-by: Parth Mannan <pmannan@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@parthmannan parthmannan requested a review from ananthsub April 10, 2026 00:02
@coderabbitai

coderabbitai Bot commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 63889bb0-cc46-4826-857b-4340971a43da

📥 Commits

Reviewing files that changed from the base of the PR and between bb767c9 and 7c739ef.

📒 Files selected for processing (1)
  • src/megatron/bridge/training/initialize.py
📝 Walkthrough

Walkthrough

A single parameter hybrid_context_parallel is forwarded from the model configuration to the parallel_state.initialize_model_parallel() function call in the distributed initialization function.

Changes

Cohort / File(s) Summary
Distributed Initialization Configuration
src/megatron/bridge/training/initialize.py		 Added forwarding of hybrid_context_parallel=model_config.hybrid_context_parallel parameter to initialize_model_parallel() call in _initialize_distributed.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Results For Major Changes ⚠️ Warning PR adds Hybrid CP support without adequate testing documentation or unit test coverage for the new parameter. Update PR description with test results. Add unit tests validating hybrid_context_parallel parameter passing. Include regression testing documentation.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically identifies the main change: adding a Hybrid CP argument to the initialize_model_parallel function, which matches the code modification in the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch pmannan/hybrid_cp_mlm_init

Comment @coderabbitai help to get the list of available commands and usage tips.

@yaoyu-33 yaoyu-33 merged commit 2bb907a into main Apr 10, 2026
2 checks passed
@yaoyu-33 yaoyu-33 deleted the pmannan/hybrid_cp_mlm_init branch April 10, 2026 02:07
yaoyu-33 pushed a commit to conver334/Megatron-Bridge that referenced this pull request Apr 10, 2026
@parthmannan @yaoyu-33
Add Hybrid CP arg to initialize_model_parallel (NVIDIA-NeMo#3259)
815f159 
Signed-off-by: Parth Mannan <pmannan@nvidia.com>
@cuichenx cuichenx mentioned this pull request May 8, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaoyu-33 yaoyu-33 yaoyu-33 approved these changes

@ananthsub ananthsub Awaiting requested review from ananthsub

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@parthmannan @yaoyu-33