LGTM

btw, I have one small question, why switch the max_num_tokens from 1 to 0?

btw, I have one small question, why switch the max_num_tokens from 1 to 0?

Did you mean switch from 0 to 1? This is because our default logic for generation model suppose it will generate 1 token after prefill. Although embedding models do not generate tokens, I added one dummy token to cheat with the shared part of the code.

btw, I have one small question, why switch the max_num_tokens from 1 to 0?

Did you mean switch from 0 to 1? This is because our default logic for generation model suppose it will generate 1 token after prefill. Although embedding models do not generate tokens, I added one dummy token to cheat with the shared part of the code.

I see, thanks for answering