Hello!

The issue that you describe (i.e. truncating cutting off chat template suffixes) was a recurring issue that I tried to tackle in a few ways when introducing the new CrossEncoder implementation. I tried a lot of different solutions, but nothing worked elegantly. I ended up asking vLLM how they tackle it, and they mention that they let truncation happen if it happens. Your proposal here is also roughly one of the ideas I had (detect tail, tokenize tail, lower max-length by tail length, preprocess inputs, add tail tokens), but it was very messy.

I tried a lot of approaches on the tokenizer side as well, e.g. make the tail part of an automatic suffix so the normal self.processor.apply_chat_template works out of the box, but tokenizers didn't support that either. I asked the transformers team as well, and they don't really have an elegant solution for this either, but they don't necessarily need one as the processing/tokenizing is the users' responsibility (unlike in ST).

I think it's important to revisit this, but I'm not sure what the best fix is yet.

• Tom Aarsen

Okay, I pushed some follow-up changes in https://github.com/huggingface/sentence-transformers/tree/pr-3787 (4693ae2 in particular), please have a look, I'm curious what you think.

In short, I'm revisiting the approach, but now by separately tokenizing the tail (with caching) and replacing the last non-padding tokens of every batch item with the tail. It seems to work pretty well, although I'm still not the biggest fan.

• Tom Aarsen

Hello!

Thank you for taking a look and for pushing the follow-up implementation.

I think your version is much more general than my rather ad-hoc initial approach. In particular, automatically deriving the chat template suffix instead of relying on a fixed number of final tokens should make this work better across other causal LM rerankers and related last-token use cases.

As you mentioned, a really clean solution probably requires support at a lower level, such as in tokenizers / transformers, so that the content can be truncated while preserving the chat template suffix explicitly. But given the current APIs, I think your post-processing approach is a very reasonable way for Sentence Transformers to absorb this issue.

From a user perspective, I think this is quite valuable. Without this kind of handling, reranking scores can silently become much worse once the formatted input crosses max_length, and that failure mode is not obvious. I only noticed it when benchmarking with max_length=2048 the scores looked unexpectedly bad for some longer inputs, and it took some digging to realize that the assistant/scoring suffix had been truncated away.

So even if the solution is not perfectly elegant, I think it addresses a real and hard-to-notice problem for Sentence Transformers users of causal LM rerankers.

Thanks again for the thoughtful explanation and implementation. I really appreciate the care you put into this.

I think we're on the same page here. The truncation is a bigger issue than I was giving it credit for. It's also very possible that my https://huggingface.co/datasets/cross-encoder/ettin-reranker-v1-data data is wrong for the larger inputs (if anything exceeded 8k?) due to this. I'll try and continue working on https://github.com/huggingface/sentence-transformers/tree/pr-3787 to get it shipped in a minor release (I think I can justify that, i.e. not having to wait until a major release).

• Tom Aarsen

Thank you, that sounds great.

I agree that this is important enough to be worth handling in a minor release if you think it fits the release policy. The failure mode is quite subtle from the user side, so having Sentence Transformers handle it directly would be very helpful.

Thanks as always for looking into this so carefully and for moving it forward.

If it's okay with you, I'll bring my changes into this PR as it was already built on your proposals.

Pull Request overview

• Preserve the trailing chat-template suffix (e.g. the assistant prefill) when max_length truncation would otherwise cut it off.
• Lives in the base Transformer, so it applies across SentenceTransformer, CrossEncoder and SparseEncoder.

Details

Causal LM rerankers like Qwen/Qwen3-Reranker-0.6B don't score a query-document pair from a CLS token or mean pooling; they read the logits at the final token position and compare the yes/no next-token logits. Last-token-pooling embedders are the same, reading the final hidden state. For either to work, the formatted input has to keep the tail of the chat template - the assistant prefill, a trailing EOS, etc.

The problem is that when max_length is set and the pair is long, apply_chat_template(..., truncation=True, max_length=...) truncates from the right and removes that tail. The input still looks like a valid (truncated) prompt, but the final position is now somewhere in the middle of the document, so the score or embedding is read from the wrong place and can collapse - a clearly relevant pair suddenly scores very low. With Qwen3-Reranker-0.6B, max_length=128 and a relevant ~200 token pair, the raw score dropped to around -0.8; with this PR it returns to the +6 to +9 range, in line with the untruncated score.

from sentence_transformers import CrossEncoder
import torch

# query + a relevant long document, formatted length > max_length
model = CrossEncoder("Qwen/Qwen3-Reranker-0.6B", max_length=128, activation_fn=torch.nn.Identity())
model.predict([[query, long_document]])[0]
# before: ~ -0.8   (final position landed mid-document)
# after:  ~ +7     (assistant prefill restored)

@hotchpotch originally tackled this with a head/tail token-reservation approach; I've reworked it to derive the suffix automatically, run on both the tokenizer and ProcessorMixin paths, and handle last-token-pooling embedders, not just causal-LM rerankers. I tried a few ways to handle it during tokenization, but apply_chat_template renders to a flat string that the tokenizer then truncates blind to the template structure, and neither tokenizers nor transformers exposes a way to trim only the content while keeping the suffix. So this runs post-hoc: truncation happens as usual, and afterwards I splice the chat-template suffix back onto any row whose tail was cut. Rows that fit already end with the suffix and are left untouched, as is truncation=False.

The suffix is derived automatically rather than hard-coded per model: I render the conversation's role layout twice with two different fillers and take the longest common token suffix, which is exactly the run of tokens the template appends after the content. It's cached (LRU-bounded), works for both the tokenizer and ProcessorMixin paths, locates each row's real tail from the attention mask (so it's agnostic to the padding side and pad_to_multiple_of), and is applied per row so a batch mixing different layouts is handled correctly. system messages (i.e. the prompt/instruction) are kept fixed during derivation, so an instruction that a template renders in the tail is restored too.

This only applies to text-only content. Image/audio/video placeholder tokens can't be truncated without desyncing them from their pixel/feature values (the model would error), so those rows are left as-is; text-only inputs to a multimodal model still qualify.

I also added restore_suffix in the chat template kwargs, defaulting to False. To keep the raw behaviour for a call (i.e. just standard truncation without restoration), pass processing_kwargs={"chat_template": {"restore_suffix": False}}.

• Tom Aarsen

Of course, please feel free to bring your changes into this PR. Your implementation is much more robust and general than my initial version, so I’d be very happy to have this PR updated with that approach.

Thank you for merging the PR!