Non-Record Submission: True Discrete Diffusion Model

Answers OpenAI's explicit request for text diffusion submissions. This is a genuine discrete diffusion model, not an AR model with diffusion-inspired loss.

Approach

• Clean prefix conditions a diffused suffix via absorbing-mask corruption
• Timestep embeddings and prefix/suffix role embeddings inform the model
• Denoising loss computed only on corrupted suffix tokens
• Evaluation via approximate prefix-conditioned diffusion pseudo-log-likelihood (PLL)

Results (8xH100 SXM, 600s)

• diffusion_pll_bpb: 1.8587 (approximate, not directly comparable to AR BPB)
• Steps: 2,398 at 250ms/step
• Artifact: 13.7MB (under 16MB)
• Training loss: 41.2 → 4.84 (converging but not converged)

Why This Matters

This is (to our knowledge) the first true discrete diffusion submission in the competition. The PLL evaluation is approximate and not directly comparable to AR BPB, but it demonstrates that discrete diffusion can be trained within the competition's constraints.

Key Limitations

• Approximate PLL, not exact AR BPB — scores are not directly comparable
• 250ms/step is slow (no torch.compile, no FA3)
• Model would benefit significantly from longer training (unlimited compute track)

Credits

• Discrete diffusion: MDLM (Sahoo et al.), SEDD (Lou et al.)
• Baseline architecture: OpenAI Parameter Golf starter code