@merrymercy, I re-added the repetition penalty that was removed by #3988. I found this issue when comparing the sglang output with HF output when the repetition penalty was applied. Please help review it. Thank you.

Why do you need this? OpenAI API does not provide this functionality. Why are the frequency and presence penalty not enough?

@merrymercy , I do offline generation for one of my use cases, which wants to align with HF output, and another use case mainly uses Python Request. Do you mean if I use the frequency and presence penalty, I can get the same behavior with the repetition penalty method?

Can you disable repetition penalty for your HF use cases?

Can you disable repetition penalty for your HF use cases?

I can, but I think we shouldn't forbid this usage case for our customer users.

make sense. We can merge this. Can you add some test cases here?

make sense. We can merge this. Can you add some test cases here?

sglang/test/srt/test_penalty.py

Lines 62 to 70 in 094891c

	def test_frequency_penalty(self):
	self.run_decode({"frequency_penalty": 2})
	def test_min_new_tokens(self):
	self.run_decode({"min_new_tokens": 16})
	def test_presence_penalty(self):
	self.run_decode({"presence_penalty": 2})

Done.

@merrymercy can we merge it?

@merrymercy

@merrymercy could you help review it? thanks!

@merrymercy We initially used vllm for model inference, and recently we plan to integrate sglang, allowing users to choose between vllm and sglang for inference. However, since we previously set the repetition_penalty parameter when using vllm for some tasks, and currently sglang does not support this parameter. It is difficult for us to align the results between vllm and sglang. We hope to have the repetition penalty feature reintegrated to sglang. Thanks!

Is this going to be merged? HF repetition penalty and OpenAI freq/presence penalties have highly different behavior (mainly because HF rep-pen takes the whole prefill context into account, while freq/pres. penalties are only taking currently generated tokens in account) and it is quite a pain having to manually merge this for every SGLang version for our use case.

cc @merrymercy

Support this PR. For small models or those without sufficient SFT, using "anyway" to prevent the model from repeating is still quite necessary.

@XiaobingSuper Hi, Currently, repetition penalty only considers the generated text, not the original input text. Both HF Transformers and vLLM consider the original input text.

Modify as follows:

class BatchedRepetitionPenalizer(_BatchedPenalizer):

    def _prepare(self):
        batch_cumulated_repetition_penalties = []
        for req in self.orchestrator.reqs():
            cumulated_repetition_penalties_lst = [1] * self.orchestrator.vocab_size
            for idx in req.origin_input_ids:
                cumulated_repetition_penalties_lst[idx] = req.sampling_params.repetition_penalty
            batch_cumulated_repetition_penalties.append(cumulated_repetition_penalties_lst)

        self.cumulated_repetition_penalties = torch.tensor(
            data=batch_cumulated_repetition_penalties,
                dtype=torch.float32,
                device=self.orchestrator.device,
        )

        self.repetition_penalties = (
            torch.tensor(
                data=[
                    req.sampling_params.repetition_penalty
                    for req in self.orchestrator.reqs()
                ],
                dtype=torch.float32,
                device=self.orchestrator.device,
            )
        ).unsqueeze_(1)

Inactive and duplicate #21258