Checklist
Describe the bug
Using SG_DEEPGEMM_JIT=1 with data parallel attention triggers OOM
- Hardware: H200
- Docker image:
lmsysorg/sglang:v0.4.5.post3-cu121
- Model: DeepseekV3
Reproduction
docker run --rm -it \
--runtime nvidia --gpus all --ipc host --privileged --ulimit memlock=-1 --ulimit stack=67108864 \
-e SG_DEEPGEMM_JIT=1 \
lmsysorg/sglang:v0.4.5.post3-cu121 \
python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --host 0.0.0.0 --port 8000 --tp 8 --trust-remote-code \
--enable-dp-attention --dp-size 8
Environment
Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 570.124.06
PyTorch: 2.6.0+cu124
sglang: 0.4.5.post3
sgl_kernel: 0.0.9.post2
flashinfer: Module Not Found
triton: 3.2.0
transformers: 4.51.1
torchao: 0.10.0
numpy: 2.2.5
aiohttp: 3.11.18
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.30.2
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.16
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.3
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.17
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.49.0
litellm: 1.67.0.post1
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-175 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-175 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-175 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-175 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 0-175 0 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 0-175 0 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 0-175 0 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 0-175 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
Hypervisor vendor: KVM
ulimit soft: 1048576
Checklist
Describe the bug
Using
SG_DEEPGEMM_JIT=1with data parallel attention triggers OOMlmsysorg/sglang:v0.4.5.post3-cu121Reproduction
Environment
Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H200 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 570.124.06 PyTorch: 2.6.0+cu124 sglang: 0.4.5.post3 sgl_kernel: 0.0.9.post2 flashinfer: Module Not Found triton: 3.2.0 transformers: 4.51.1 torchao: 0.10.0 numpy: 2.2.5 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.30.2 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.16 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.3 multipart: Module Not Found zmq: Module Not Found uvicorn: 0.34.2 uvloop: 0.21.0 vllm: Module Not Found xgrammar: 0.1.17 openai: 1.75.0 tiktoken: 0.9.0 anthropic: 0.49.0 litellm: 1.67.0.post1 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-175 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-175 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-175 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-175 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 0-175 0 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 0-175 0 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 0-175 0 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 0-175 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks Hypervisor vendor: KVM ulimit soft: 1048576