Feature: Support of Manifold Hyper Connection(mHC) by jingqiny-99 · Pull Request #3430 · NVIDIA/Megatron-LM

jingqiny-99 · 2026-02-14T06:59:17Z

What does this PR do ?

mirror PR of #2943

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

copy-pr-bot · 2026-02-14T06:59:24Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…onnection(mHC). (NVIDIA#2943) Co-authored-by: Jingqin Yang <jingqiny@login-eos01.eos.clusters.nvidia.com> Co-authored-by: root <root@eos0478.eos.clusters.nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

jingqiny-99 · 2026-04-01T05:55:40Z

/claude review

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

Phlip79 · 2026-05-12T01:06:37Z

Closing this PR in favor of #4531.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

… manifold hyper connection Adds the core Manifold Hyper Connection (mHC) module and the supporting transformer-block / transformer-layer / config / recompute changes. This is the first split of NVIDIA#3430 (mirror of upstream NVIDIA#2943 + kernel-fusion NVIDIA#3828), covering only files owned by core-adlr / core-nemo / transformer / cuda-graphs reviewers. GPT model wiring, pipeline-parallel support, fused mHC cuTile kernels, and the functional-test recipe are deferred to follow-up split PRs. Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943, basic pytorch impl only — kernel fusion NVIDIA#3828 deferred). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

CI run #25129636936 (job tests/unit_tests/models/**/*.py) failed with "Mamba MoE config drift detected!" because PR NVIDIA#1 adds five new TransformerConfig fields (enable_hyper_connections, mhc_init_gating_factor, mhc_recompute_layer_num, mhc_sinkhorn_iterations, num_residual_streams) but this PR previously deferred the corresponding test_mamba_moe_model.py GOLDEN_CONFIG update (the file has since been renamed to test_hybrid_moe_model.py on main). This is a generic config-drift sentinel — any PR that adds TransformerConfig fields must update this golden snapshot — so it belongs in this split, not in a deferred follow-up. Add the five keys with the same default values used in the original PR NVIDIA#3430 / upstream NVIDIA#2943.

… forward The Codecov patch-coverage gate on NVIDIA#4531 fails (68.25% < 80% target) because two new code paths in this split lack any test: * fused_bias_dropout._get_checkpointed_bda — only reachable via the mHC layer's recompute path, which the unit-test lane doesn't exercise. * transformer_block._build_mhc_recompute_layer_plan, _finalize_mhc_recompute_layer, and the input_expand / output_contract calls in TransformerBlock.forward — only reachable from a full block forward with mHC selective recompute enabled. Neither original test exists in PR NVIDIA#3430: * No test in the upstream PR references get_bias_dropout_add or _get_checkpointed_bda. * The block-level mHC forward test in NVIDIA#3430 (TestPPForwardWithMHC) is multi-GPU-only (torchrun --nproc-per-node=2+) and so wouldn't run in the codecov-feeding unit-test lane even if lifted. Add direct unit coverage: tests/unit_tests/fusions/test_bias_dropout_fusion.py TestBiasDropoutAddMhcRecompute test_checkpointed_bda_forward_backward (parametrized fused/with_bias) test_checkpointed_bda_chained_managers test_get_bda_without_manager_unchanged tests/unit_tests/transformer/test_mhc_block_manager.py TestTransformerBlockMHCRecompute test_recompute_plan_no_layer_num test_recompute_plan_with_layer_num test_recompute_plan_disabled test_block_forward_input_expand_output_contract Combined, these directly hit ~80 of the previously-missed lines in fused_bias_dropout.py (~14 of 19 missed) and transformer_block.py (~12 of 16 missed), which should bring patch coverage above the 80% threshold.

@jingqiny-99

Second split of NVIDIA#3430 (mirror of upstream NVIDIA#2943). Wires the mHC core module added in Split 1 (NVIDIA#4531) into the GPT layer specs and the training initialization path: - megatron/core/models/gpt/gpt_layer_specs.py: add enable_hyper_connection kwarg; build mHC-aware ModuleSpec selecting HyperConnectionTransformerLayer - megatron/core/models/gpt/experimental_attention_variant_module_specs.py: forward the kwarg through experimental specs - gpt_builders.py: thread enable_hyper_connection from config to spec - megatron/training/initialize.py: enable NVTX range profiling early - tests/unit_tests/models/test_gpt_layer_specs.py: coverage for the enable_hyper_connection branch - tests/unit_tests/test_fp8_param.py: minor mHC-related test wiring Depends on NVIDIA#4531 — this PR is mergeable only after Split 1 lands. Reviewer groups touched: core-adlr, core-nemo, gpt, training-adlr, training-nemo. Subsequent splits (still pending Split 1's merge): - Split 3: pipeline-parallel support - Split 4: fused mHC cuTile kernels - Split 5: functional-test recipe Original work by @jingqiny-99 in NVIDIA#3430 (upstream NVIDIA#2943). Co-authored-by: jingqiny-99 <jingqiny@nvidia.com> Co-authored-by: Dennis Liu <denliu@nvidia.com>

jingqiny-99 force-pushed the jingqiny/feature-mHC branch from a1d8ca5 to 1e66788 Compare February 14, 2026 07:10

copy-pr-bot Bot temporarily deployed to nemo-ci February 14, 2026 07:11 Inactive

copy-pr-bot Bot temporarily deployed to test February 14, 2026 07:12 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci February 14, 2026 07:48 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci February 14, 2026 08:04 Inactive

copy-pr-bot Bot temporarily deployed to test February 14, 2026 08:05 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci February 14, 2026 08:32 Inactive

yanring mentioned this pull request Feb 24, 2026

[dev] feat(mHC): Add basic pytorch implementation of manifold hyper connection(mHC). #2943

Merged

10 tasks

copy-pr-bot Bot temporarily deployed to test February 28, 2026 03:46 Inactive

copy-pr-bot Bot temporarily deployed to test February 28, 2026 06:16 Inactive

copy-pr-bot Bot temporarily deployed to test February 28, 2026 08:14 Inactive

copy-pr-bot Bot temporarily deployed to test February 28, 2026 10:21 Inactive

copy-pr-bot Bot temporarily deployed to test March 2, 2026 01:46 Inactive

copy-pr-bot Bot temporarily deployed to test March 2, 2026 03:49 Inactive

copy-pr-bot Bot temporarily deployed to test March 2, 2026 06:55 Inactive

copy-pr-bot Bot temporarily deployed to test March 3, 2026 01:22 Inactive

copy-pr-bot Bot temporarily deployed to test March 4, 2026 10:48 Inactive

copy-pr-bot Bot temporarily deployed to test March 6, 2026 02:23 Inactive

jingqiny-99 and others added 2 commits March 31, 2026 15:50

[dev] mHC kernel fusion (NVIDIA#3828)

62ea06f

jingqiny-99 force-pushed the jingqiny/feature-mHC branch from e7e1a13 to 62ea06f Compare April 1, 2026 05:48

Merge branch 'main' into jingqiny/feature-mHC

08c1a50

This was referenced May 1, 2026

Add mHC support for HybridModel on dsv4 #4567

Closed

Add mHC support for HybridModel on dsv4 #4568

Closed

Phlip79 closed this May 12, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Support of Manifold Hyper Connection(mHC)#3430

Feature: Support of Manifold Hyper Connection(mHC)#3430
jingqiny-99 wants to merge 10 commits into
NVIDIA:mainfrom
jingqiny-99:jingqiny/feature-mHC

jingqiny-99 commented Feb 14, 2026

Uh oh!

copy-pr-bot Bot commented Feb 14, 2026

Uh oh!

jingqiny-99 commented Apr 1, 2026

Uh oh!

Phlip79 commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jingqiny-99 commented Feb 14, 2026

What does this PR do ?

Contribution process

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot Bot commented Feb 14, 2026

Uh oh!

jingqiny-99 commented Apr 1, 2026

Uh oh!

Phlip79 commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

(Step 1): Add PR label `Expert Review`