Skip to content

ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies#4905

Merged
balasaajay merged 4 commits into
NVIDIA:mainfrom
balasaajay:allow-fail-nightly
May 26, 2026
Merged

ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies#4905
balasaajay merged 4 commits into
NVIDIA:mainfrom
balasaajay:allow-fail-nightly

Conversation

@balasaajay

@balasaajay balasaajay commented May 21, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Add allow_failure flag to gpt and moe recipes that are failing in nightlies.
Needed until we have a fix for #4903

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

@copy-pr-bot

copy-pr-bot Bot commented May 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@balasaajay balasaajay marked this pull request as ready for review May 21, 2026 00:16
@balasaajay balasaajay requested a review from a team as a code owner May 21, 2026 00:16
@svcnvidia-nemo-ci svcnvidia-nemo-ci requested a review from a team May 21, 2026 00:16
chtruong814
chtruong814 reviewed May 21, 2026
View reviewed changes

@chtruong814 chtruong814 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a question

Comment thread tests/test_utils/python_scripts/generate_jet_trigger_job.py Outdated
@balasaajay balasaajay requested a review from chtruong814 May 21, 2026 20:59
@balasaajay

balasaajay commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e2ac8a0)

@balasaajay

balasaajay commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e2ac8a0

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label May 21, 2026
ko3n1g
ko3n1g approved these changes May 26, 2026
View reviewed changes

@ko3n1g ko3n1g left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@balasaajay

balasaajay commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e7ce32e

@balasaajay balasaajay enabled auto-merge May 26, 2026 16:44
@balasaajay balasaajay added this pull request to the merge queue May 26, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 26, 2026

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26466289035

Merged via the queue into NVIDIA:main with commit 859b719 May 26, 2026
78 checks passed
@balasaajay balasaajay deleted the allow-fail-nightly branch May 26, 2026 18:47
Victarry added a commit to yanring/Megatron-LM that referenced this pull request May 27, 2026
@Victarry
Merge remote-tracking branch 'origin/main' into pr3431-conflict-fix
535f9d9 
* origin/main: (50 commits)
  Drain predecessor reduce-scatter at dispatch time (NVIDIA#4940)
  ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies (NVIDIA#4905)
  fix(tests): initialize num_microbatches calculator in vision cudagraph tests (NVIDIA#4986)
  test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (NVIDIA#4985)
  ci: Add support for MBridge job gating based on PR labels  (NVIDIA#4926)
  test(ci): re-enable 8experts2parallel_multi_dist_optimizer_instances_1node (NVIDIA#4984)
  test: re-enable paged stashing MoE tests (NVIDIA#4978)
  Fix elastification unwrap_model import (NVIDIA#4972)
  Avoid offsetting functional test master port (NVIDIA#4973)
  test: enable NVTE_CUTEDSL_FUSED_GROUPED_MLP via pytest fixture (NVIDIA#4931)
  chore(beep boop 🤖): Bump  (main) (2026-05-25)
  test(release): add release goldens for deepseekv3/nemotron3 and set tp2pp2 exit-interval (NVIDIA#4932)
  Fix `get_batch` return order to ignore BlendedDataset provenance fields (NVIDIA#4952)
  ci: restore perf test torchrun logs (NVIDIA#4951)
  Various training utils (NVIDIA#4872)
  ci: Update training script paths in BERT and T5 (NVIDIA#4939)
  [MXFP8/FP4-param-gather] Post processing after forced param AG in eval (NVIDIA#4562)
  Fix mxfp8 param gather numerical issue when DP overlap is off (NVIDIA#4800)
  Add TEFusedDenseMLP for Dense+Grouped GEMM fusion on SM100+ (NVIDIA#4318) (NVIDIA#4786)
  Fix paged stashing test submodules lookup (NVIDIA#4925)
  ...

# Conflicts:
#	megatron/training/training.py
janEbert pushed a commit to janEbert/Megatron-LM that referenced this pull request Jun 2, 2026
@balasaajay @janEbert
ci: Add allow_failure flag to gpt and moe recipes that are failing in…
41c823f 
… nightlies (NVIDIA#4905)
mathemakitten pushed a commit to mathemakitten/Megatron-LM that referenced this pull request Jun 12, 2026
@balasaajay @mathemakitten
ci: Add allow_failure flag to gpt and moe recipes that are failing in…
89d14ba 
… nightlies (NVIDIA#4905)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ko3n1g ko3n1g ko3n1g approved these changes

@chtruong814 chtruong814 chtruong814 approved these changes

Assignees

No one assigned

Labels

Approved All necessary approvals have been made complexity: low

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@balasaajay @svcnvidia-nemo-ci @ko3n1g @chtruong814