Skip to content

[model]Add Qwen3‑Omni training support #3317

Merged
cuichenx merged 49 commits into
NVIDIA-NeMo:mainfrom
hbhflw2000:omni3-train
May 12, 2026
Merged

[model]Add Qwen3‑Omni training support #3317
cuichenx merged 49 commits into
NVIDIA-NeMo:mainfrom
hbhflw2000:omni3-train

Conversation

@hbhflw2000

@hbhflw2000 hbhflw2000 commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Add Qwen3‑Omni training and example scaffolding with smoke conversion/inference, plus a new L0 CI launcher for the Qwen3‑Omni conversion test.

Changelog

  • Add Qwen3‑Omni example scripts and README (conversion, inference, local training) with repo‑local workspace defaults.
  • Add Qwen3‑Omni smoke conversion and inference flow (HF → Megatron → HF export).
  • Add L0 CI entrypoint for Qwen3‑Omni conversion tests with repo‑root coverage paths.
  • Add Qwen3‑Omni unit and functional tests (recipe registry, conversion, smoke validation).

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • This is PR2; PR1 is currently under review: [model] Add initial Qwen3-Omni support in Megatron-Bridge #2831
  • This PR is independent of PR1 and focuses on example/test scaffolding only. It does not introduce changes to the core model implementation.
  • The training recipe has been validated on 4 nodes × 8 GPUs with initial performance tuning.
  • Follow-up work: performance/efficiency optimizations will be handled in a separate PR.
  • Related to # (issue)

Summary by CodeRabbit

New Features

  • Qwen3-Omni Model Integration: Added full support for Qwen3-Omni multimodal model with text, image, video, and audio capabilities.
  • Checkpoint Conversion Tools: New utilities to convert checkpoints between Hugging Face and Megatron formats.
  • Training Examples: Added example scripts and recipes for fine-tuning the model with multimodal data.
  • Inference Workflows: New example scripts demonstrating single-rank and distributed inference scenarios.

Documentation

  • Added comprehensive model documentation including supported features and known limitations.
  • Added training plan documentation outlining staged implementation roadmap.
  • Added README files documenting example usage and workspace setup.

Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: hbhflw2000 <417911774@qq.com>
@yuekaizhang

Copy link
Copy Markdown
Contributor

/ok to test c8f97ed

@yuekaizhang

Copy link
Copy Markdown
Contributor

/ok to test c8f97ed

@yuekaizhang

Copy link
Copy Markdown
Contributor

/ok to test c8f97ed

@yuekaizhang

Copy link
Copy Markdown
Contributor

/ok to test 61d071b

@cuichenx cuichenx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cuichenx cuichenx merged commit 7a80c00 into NVIDIA-NeMo:main May 12, 2026
95 of 99 checks passed
gautham-kollu pushed a commit that referenced this pull request May 12, 2026
Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Co-authored-by: Huy Vu <86480512+huvunvidia@users.noreply.github.com>
Co-authored-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
vasunvidia pushed a commit to vasunvidia/Megatron-Bridge that referenced this pull request Jun 10, 2026
Signed-off-by: Lianglipeng <lianglipeng@didiglobal.com>
Signed-off-by: hbhflw2000 <417911774@qq.com>
Co-authored-by: Huy Vu <86480512+huvunvidia@users.noreply.github.com>
Co-authored-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic community-request feature New capabilities, enhancements, or enablement work ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants