Returning true for GGML_OP_GET_ROWS in offload_op will cause the token embeddings to be copied to VRAM, which is almost never worth it since this is a big tensor and this op can be run very cheaply on the CPU. I imagine that RWK uses get_rows in some way that might make it worthwhile copying the weight to VRAM in that case, and that's why @uniartisan saw a speedup, but it needs to be done in a more selective way.

@Alcpz
Which GPU do you test with?

The PR #10133 has no impact to Intel Arc 770 for llama2-7b-q4 and Meta-Llama-3-8B.Q8_0.gguf.

I´ve tested multiple GPUs. The description has data for a Nvidia A100, but I also tested on an Arc 770 and a Data Center GPU Max 1100. For these two models I see regression in performance, though I'm using Meta-Llama-3.1-8B-Instruct-Q8_0.gguf.

See below additional performance information:

build: fab5d30 (4143)

build: f4c4ce3 (this pr)

build: fab5d30 (4143)

build: f4c4ce3 (this pr)

@NeoZhangJianyu do you mean no regression during decoding phase?

@NeoZhangJianyu do you mean no regression during decoding phase?

I just test the model files as end to end.
Not find the performance change .

@uniartisan
How about your idea? Since you are the author of PR #10133

@NeoZhangJianyu I assure you, this is a significant performance problem and needs to be fixed as soon as possible. It's hard to tell why you cannot reproduce this without more details about how you are testing.

@NeoZhangJianyu you mentioned testing with Meta-Llama-3-8B.Q8_0.gguf while we are using Meta-Llama-3.1-8B-Instruct-Q8_0.gguf. Could that explain why you are not seeing the same performance drop?

My test ignore the impact to "pp512".