@DarkLight1337 @ywang96 Thanks for all the fixes!

For V1, we currently do not strictly follow the original attention in Gemma 3. The model still generates reasonable outputs, but this needs to be fixed to get the full accuracy

Hi @WoosukKwon, can you explain the main difference between the two approaches ? Thanks :-)

do we know that when the docker latest image will published?

The next release is very soon: https://github.com/vllm-project/vllm/milestone/1

How can I run Gemma 3 with a backend different from Xformers? It works very slowly. And it doesn't start with flashinfer.

How can I run Gemma 3 with a backend different from Xformers? It works very slowly. And it doesn't start with flashinfer.

Having the same problem

Can you try setting VLLM_USE_V1=0 to enable more backends?

@DarkLight1337

and

The error message shows that your --max-model-len is too high

@DarkLight1337 I understand that perfectly, but the Gemma3 model has a context window: Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size.

I think here the max model len corresponds to the sliding window length, not the total window length

In general, try to run the model with any other backend and you'll see that it doesn't work, while Xformers is terribly slow.

When will v1 support gemma3?

It's supported if you install from main branch, but there might be correctness issues because their attention mask is not fully implemented in V1.

I think here the max model len corresponds to the sliding window length, not the total window length

--max-model-len is model total context size, see https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html

There are some problems with gemma 3 support currently. V0 doesn't support flashattention and context size is huge. And with V1 I wasn't able to load gptq quant.

Edit: I was able to load gpt quant, but the context size is very low(<8192) in 24gb gpu
Edit: I was able to load 8k context with max-num-seq 10 and --gpu-memory-utilization 0.99, though I expected more context to fit since model is 16gb, so there should be 8 gb of free vram for context.

Edit: I was able to load gpt quant, but the context size is very low(<8192) in 24gb gpu
Edit: I was able to load 8k context with max-num-seq 10 and --gpu-memory-utilization 0.99, though I expected more context to fit since model is 16gb, so there should be 8 gb of free vram for context.

Could you kindly share a code snippet? I am facing few issues too and that would help a ton! @anunknowperson 🙏

@pietrobolcato

My launch string for openai server is

CUDA_VISIBLE_DEVICES=0 vllm serve ISTA-DASLab/gemma-3-27b-it-GPTQ-4b-128g --max-model-len 8192 --max-num-seqs 10 --gpu-memory-utilization=0.99

Probably you can get params from here for code. Engine is V1, V0 context is too big.

@DarkLight1337 @WoosukKwon
could you please check this issue related to Gemma3-AWQ?