registry.py:321] from vllm.attention.backends.flash_attn import FlashAttentionMetadata ERROR 02-21 11:17:10 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/flash_attn.py", line 25, in <module> ERROR 02-21 11:17:10 registry.py:321] from vllm.vllm_flash_attn import (flash_attn_varlen_func, ERROR 02-21 11:17:10 registry.py:321] ImportError: cannot import name 'flash_attn_varlen_func' from 'vllm.vllm_flash_attn' (unknown location) ERROR 02-21 11:17:10 registry.py:321] Traceback (most recent call last): File "/usr/local/bin/vllm", line 8, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
It looks like the problem is caused by importing FlashAttentionMetadata in MambaMixer2, which pulls in vllm_flash_attn, which is unsupported on RoCM.
Your current environment
Via @hackey:
🐛 Describe the bug
See #6479 (comment)
Specifically this part:
It looks like the problem is caused by importing FlashAttentionMetadata in MambaMixer2, which pulls in vllm_flash_attn, which is unsupported on RoCM.
Before submitting a new issue...