I tested the P2 loss (1-p)^2 training approach on our stack (11L, 512d, Full Hessian GPTQ, XSA-all, BigramHash) and wanted to flag a potential metric issue.

The concern: The P2 loss reweights the cross-entropy during training, focusing gradients on hard tokens. This is a legitimate training technique. However, if the same P2-weighted loss is also used during evaluation (via model.forward() returning the P2-weighted loss), the reported val_bpb is not standard cross-entropy — it's a reweighted metric that will be lower than the true BPB.

Per the README and Issue #1017:

"val_bpb is the prequential code length of a causal predictor"

This requires standard cross-entropy evaluation (Condition 2: full normalized distribution scored with log p(x_t)), not a reweighted variant.

My test results:

When evaluated with standard F.cross_entropy (the correct metric), the P2-trained model scores 1.157 BPP — significantly worse than standard CE training (1.111). The P2 loss sacrifices easy-token accuracy to focus on hard tokens, which nets worse overall BPP.

The reported 1.0577 BPP may be using the P2-weighted loss path for evaluation rather than standard cross-entropy. Could you confirm whether eval_val() uses model.forward() (P2-weighted) or F.cross_entropy(model.forward_logits()) (standard CE)?

cc @0hq @valerio-oai

Thanks for the careful review. You are correct. Validation is currently using the P2-weighted loss via model.forward().

I will update the evaluation to use standard cross-entropy (unweighted), rerun, and resubmit with corrected metrics.

Appreciate you flagging this.