test: re-enable paged stashing MoE tests#4978
Merged
ko3n1g merged 2 commits intoMay 26, 2026
Merged
Conversation
Remove flaky_in_dev markers from TestPagedStashing.test_forward_backward_4_layers and TestPagedStashingOverBudget.test_overload_factor_and_over_budget. The underlying AttributeError on transformer_layer_spec.submodules.mlp.submodules was fixed in NVIDIA#4925 by routing through get_submodules(); the skip markers added in NVIDIA#4931 are no longer needed. Closes NVIDIA#4935 Signed-off-by: oliver könig <okoenig@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test b616990 |
The tests configure fp8_recipe='mxfp8', which TE rejects on devices below compute capability 10.0 with 'Device compute capability 10.0 or higher required for MXFP8 execution'. The previous flaky_in_dev marker was masking this hardware-incompatibility on H100 CI. Replace it with an explicit SM100 skipif so the tests actually run on Blackwell and skip cleanly elsewhere. Signed-off-by: oliver könig <okoenig@nvidia.com>
Contributor
Author
|
/ok to test 05ae563 |
thomasdhc
approved these changes
May 26, 2026
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26452178663 |
santhnm2
pushed a commit
to santhnm2/Megatron-LM
that referenced
this pull request
May 26, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
Victarry
added a commit
to yanring/Megatron-LM
that referenced
this pull request
May 27, 2026
* origin/main: (50 commits) Drain predecessor reduce-scatter at dispatch time (NVIDIA#4940) ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies (NVIDIA#4905) fix(tests): initialize num_microbatches calculator in vision cudagraph tests (NVIDIA#4986) test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (NVIDIA#4985) ci: Add support for MBridge job gating based on PR labels (NVIDIA#4926) test(ci): re-enable 8experts2parallel_multi_dist_optimizer_instances_1node (NVIDIA#4984) test: re-enable paged stashing MoE tests (NVIDIA#4978) Fix elastification unwrap_model import (NVIDIA#4972) Avoid offsetting functional test master port (NVIDIA#4973) test: enable NVTE_CUTEDSL_FUSED_GROUPED_MLP via pytest fixture (NVIDIA#4931) chore(beep boop 🤖): Bump (main) (2026-05-25) test(release): add release goldens for deepseekv3/nemotron3 and set tp2pp2 exit-interval (NVIDIA#4932) Fix `get_batch` return order to ignore BlendedDataset provenance fields (NVIDIA#4952) ci: restore perf test torchrun logs (NVIDIA#4951) Various training utils (NVIDIA#4872) ci: Update training script paths in BERT and T5 (NVIDIA#4939) [MXFP8/FP4-param-gather] Post processing after forced param AG in eval (NVIDIA#4562) Fix mxfp8 param gather numerical issue when DP overlap is off (NVIDIA#4800) Add TEFusedDenseMLP for Dense+Grouped GEMM fusion on SM100+ (NVIDIA#4318) (NVIDIA#4786) Fix paged stashing test submodules lookup (NVIDIA#4925) ... # Conflicts: # megatron/training/training.py
janEbert
pushed a commit
to janEbert/Megatron-LM
that referenced
this pull request
Jun 2, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
mathemakitten
pushed a commit
to mathemakitten/Megatron-LM
that referenced
this pull request
Jun 12, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude summary
Removes the
@pytest.mark.flaky_in_devmarkers from both paged-stashing MoE tests and replaces them with an explicit SM100 (Blackwell) skip, so the tests run on Blackwell CI and skip cleanly on H100/A100 instead of failing with a TE runtime error.Why the original markers were wrong
flaky_in_devwas added in #4931 to keep CI green, but the failures were not flaky — they were a missing hardware gate. Both classes configurefp8_recipe='mxfp8'; TE rejects this on devices below compute capability 10.0 with:H100 CI is SM90, so the tests can never have run there. The previous
flaky_in_devmarker hid this fact in thedevenvironment.Change
Same pattern already used in
tests/unit_tests/distributed/megatron_fsdp/test_mcore_fully_sharded_data_parallel.py:983andtests/unit_tests/inference/test_mxfp8_utils.py:349.Affected tests
tests/unit_tests/transformer/moe/test_paged_stashing.py::TestPagedStashing::test_forward_backward_4_layerstests/unit_tests/transformer/moe/test_paged_stashing.py::TestPagedStashingOverBudget::test_overload_factor_and_over_budgetCloses #4935.