Skip to content

Add Helios-14B Video Generation Pipelines#13208

Merged
sayakpaul merged 109 commits intomainfrom
helios
Mar 4, 2026
Merged

Add Helios-14B Video Generation Pipelines#13208
sayakpaul merged 109 commits intomainfrom
helios

Conversation

@dg845
Copy link
Collaborator

@dg845 dg845 commented Mar 4, 2026

What does this PR do?

This PR adds pipelines for the Helios-14B video generation model (project page, official code, model weights). Helios is a 14B parameter video generation model which can generate minute-scale video at 19.5 FPS on a single H100 GPU. Both a full model and distilled model are available, which correspond to HeliosPipeline and HeliosPyramidPipeline respectively; each pipeline supports text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) tasks.

T2V inference example:

import torch
from diffusers import AutoModel, HeliosPipeline
from diffusers.hooks.group_offloading import apply_group_offloading
from diffusers.utils import export_to_video

vae = AutoModel.from_pretrained("BestWishYsh/Helios-Base", subfolder="vae", torch_dtype=torch.float32)

# group-offloading
pipeline = HeliosPipeline.from_pretrained(
    "BestWishYsh/Helios-Base",
    vae=vae,
    torch_dtype=torch.bfloat16
)
pipeline.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="block_level",
    num_blocks_per_group=1,
    use_stream=True,
    record_stream=True,
)

prompt = """
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue 
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with 
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, 
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades 
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and 
the vivid colors of its surroundings. A close-up shot with dynamic movement.
"""
negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=99,
    num_inference_steps=50,
    guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_base_t2v_output.mp4", fps=24)
helios_base_t2v_output.mp4

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu
@sayakpaul
@SHYuanBest

yuanshenghai and others added 30 commits February 24, 2026 14:20
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving so that @dg845 can merge it as appropriate. I guess the doc errors should be easy to fix?

@SHYuanBest
Copy link
Contributor

Thank you all for merging!

SHYuanBest and others added 2 commits March 4, 2026 20:44
* fix docs

* remove redudant

* remove redudant

* fix group offload

* Removed fixes for group offload
@sayakpaul
Copy link
Member

Will merge so as to not block the release longer. Fixes can follow in another PR if needed. Thanks so much for your contributions, @SHYuanBest!

@sayakpaul sayakpaul merged commit 33f785b into main Mar 4, 2026
31 of 33 checks passed
@sayakpaul sayakpaul deleted the helios branch March 4, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants