This is not legal because the reported score does not come from one causal model making one prediction per token. Instead, the same evaluation sequence is traversed multiple times under different adaptation histories, and then each token is assigned the best loss it achieved across those passes. That per-token min is an oracle hindsight selector: it uses repeated exposure to the test data to decide, after the fact, which trajectory each token should get credit from. Even if each individual pass is locally causal, the aggregate scoring rule is not, so the result overstates real predictive performance and exploits the evaluation protocol rather than measuring a deployable model.

Does the min(NLL) per token require knowing the outcome of all three passes before selecting a score?

@andrew-medrano Thanks for the feedback, I think the key point here is that the scoring rule is structurally similar to sliding window evaluation which is used by every competitive submission and explicitly permitted by the rules. In sliding window with stride=64 each token is scored around 32 times under different context windows, and the final BPB uses the score from the window that gave that token the most context. The best context for each token is determined after all windows have been evaluated. This is standard practice and not considered an oracle selector because each individual score is a valid causal prediction. Multi-pass TTT works the same way. Each token is scored multiple times under different adaptation states, and the final BPB uses the best score. Each individual score is a valid causal prediction made before any training on that token. The only difference is the axis of variation where sliding window varies context length and multi-pass varies adaptation history. The rules state that evaluation methods are unrestricted and that test-time training is allowed on already evaluated tokens and both conditions are satisfied in every pass independently.

To answer @saml212's question, yes the min(NLL) per token is computed after all passes complete. Each pass independently produces a valid, causal NLL for every token where tokens are scored first, then can be used for updates, with no access to future tokens. So the aggregation step operates over already computed and rule compliant outputs. This is similar to sliding window evaluation, where each token is scored multiple times under different contexts and the final score is computed after all windows complete. In this case, we vary adaptation trajectories instead of context windows. Importantly, the selection step does not influence how predictions are made it simply chooses among independently valid causal evaluations.

lol

I took a quick look. Given that you shuffle the data, how do you guarantee that a given TTT pass hasn't seen tokens later in the same document? Otherwise, it's not causal.

@Sarimsaljook Are you serious?

First, with sliding-window evaluation each token is assigned one designated score, not scored many times and then given whichever score looks best after the run. The overlap is just a bookkeeping device to ensure that every token is evaluated once with the maximum valid left context allowed by the model’s context limit. It is not a best-of-N procedure.

Second, the core problem here is the after-the-fact selection step. You run multiple passes that produce different test-induced adaptation histories, then for each token you retrospectively choose the lowest NLL across those passes. That means the reported metric does not come from one causal evaluation path, it comes from hindsight selection over multiple evaluations of the same test data. Even if each pass is individually causal, the final per-token min is still an oracle aggregation rule.

You have to commit to the prediction you are submitting before you see the NLL. Otherwise, in the extreme, you could submit one prediction for every possible next token and then, after seeing the losses, simply choose the one that got NLL = 0. That would not be evaluation of a single model prediction at all, it would just be selecting the winner after the fact.

@andrew-medrano Thanks for your feedback. The problem here is your extreme example fundamentally misunderstands how NLL is calculated. You can't "submit one prediction for every possible next token" and pick the one with NLL = 0. NLL is the negative log probability the model assigns to the actual ground truth token. To achieve a lower NLL, the model's specific causal state must legitimately assign a higher probability mass to the correct, unseen token. The min(NLL) selector does not look at different token guesses to find the right answer, it looks at different valid model states evaluating the exact same sequence. It cannot fabricate performance that the model's weights don't genuinely possess.

Regarding the requirement to "commit" to a single evaluation path, there is no such rule in this competition. The foundational guideline for the evaluation phase is explicitly stated as "Evaluation methods are unrestricted."

Whether sliding window is viewed as a "best-of-N" procedure or a "maximum valid context" bookkeeping device, both methods share the exact same underlying mechanism, using redundant inference-time compute to run multiple forward passes over the same evaluation data to achieve a better score. Sliding window uses that compute to optimize the context state. Multi-pass TTT uses that compute to optimize the adaptation weight state.

Because every individual pass in this submission is strictly causal, score-before-train, and fully rule-compliant, selecting the most accurate valid model state for a given token does not constitute an "oracle" exploit, and instead is standard trajectory ensembling, which is a legitimate and widely recognized reallocation of test-time compute.

@Sarimsaljook Using min(NLL) is invalid, end of story. You cannot choose which prediction to use after you know how good each one is.

@andrew-medrano I have literally laid out the technical reasoning across multiple comments in this thread. Also I take my work very seriously and have been actively reviewing it myself throughout this process. Sliding window evaluation scores each token multiple times under different context windows and selects the best score after all windows complete which is a universal standard in this competition. Multi-pass TTT does the same thing and the only difference is adaptation history instead of context length. The min(NLL) selector operates over fixed probability distributions that the model committed to before the correct token was revealed and it cant fabricate performance the model doesn't have. The competition rules state that evaluation methods are unrestricted and no existing ruling requires a single evaluation path per token. I am happy to elaborate but I will defer further discussion to the organizers. The code, logs, and method description are all public for review.

@rfgordan We dont shuffle tokens because every pass processes the validation set in the original sequential token order. Within each pass, each batch is scored first with forward under torch.inference_mode and the NLL recorded, then trained on. The next batch is always later in the sequence so no future tokens are seen before earlier ones in any pass. What changes between passes is the optimizers starting position meaning which chunk the AdamW begins accumulating gradients from. The token ordering within every pass is identical and strictly left to right. You can verify this in the code too the positions list is always constructed from range(rank_start, rank_end - seq_len, chunk_stride) and the shift only rotates which element of that list comes first it doesn't reorder the actual token sequence.

Agreed that this is an invalid submission due to scoring tokens, adapting their weights to these new scores, doing this multiple times and choosing the best NLL. This is the same as training on the eval set, and is disallowed. Closing for now.