Skip to content

state of Context Parallel & Flex attention #2417

@rakkit

Description

@rakkit

Hi, i am trying to catch up what's the state of Context Parallel for Flex attention.

Per #2145, CP + Flex attention is supported now (llama 3 & llama4) ? and does it means "generally" Flex attention can work with CP? For example sliding window attention?

And is this rope_cache #2077 blocked Flex attention CP for qwen & GPT-OSS and/or is there any another reasons?

what aboutdeepseek?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions