The lint check's whitelist guard requires every file with zero pyrefly
errors to appear in project-includes. This PR adds new x_token, arrow_text,
and tools/x_token files that are zero-error but were not yet whitelisted,
so the guard failed (it reports only the first offender because the step
runs under bash -e). Add all six to the whitelist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

get_topk_projection's cache-miss path returned a freshly-built
(indices, likelihoods) tuple instead of the one stored in
_TOPK_PROJECTION_CACHE, so the first (miss) call and subsequent (hit)
calls handed back different tuple objects. This broke
test_repeat_call_hits_cache's identity assertion (a is b). Store and
return the same tuple.

Also update test_dtensor_hsdp_dispatches_distinct_batches: Policy.train
now threads check_dim_skip_keys through common_kwargs (added in this PR
for cross-tokenizer skip-keys plumbing), so the expected
run_all_workers_sharded_data call must include 'check_dim_skip_keys': None.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <adithyakvh@gmail.com>

clean_model_name_for_filename strips parameter counts (digits + B/M) and
known suffixes; it correctly keeps model-family version numbers, so
'microsoft/phi-4-Base' -> 'microsoft/phi-4'. The assertion expected
'microsoft/phi', which also contradicts the same test's first case that
keeps the 'Llama-3.2' version. Correct the expected value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <adithyakvh@gmail.com>

…fresh prep docs

Address review comments on PR NVIDIA-NeMo#2508:
- cross_tokenizer_collate: pin tokenizer padding_side="right" so
  input_lengths / [:length] slicing and token-chunk alignment stay
  correct for tokenizers that default to left-padding.
- xtoken setup: drop hidden .get() defaults in the backend-gate asserts
  (_v2 -> .get("_v2"); tensor/context_parallel_size -> direct reads) per
  config-conventions.
- get_full_logits_ipc: synchronize after copy_ into the persistent IPC
  buffer so the consumer can't read a partially written buffer.
- xtoken train loop: refresh the stale release_ipc_buffer comment (it is
  a no-op under the persistent-buffer design).
- build_projection_matrix.sh: renumber prep steps to match the guide
  (seed + Steps 1-3 instead of 1/2/3/4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

The wrapper unconditionally ran minimal_projection_generator as a seed
pass and fed its output into the multitoken step, so the one-command
Quickstart produced a different matrix than the documented 3-step recipe.
Start the chain at Step 1 (minimal_projection_via_multitoken) building
from scratch. minimal_projection_generator.py is left in place for manual
use; only the wrapper stops invoking it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

Replace the results curves/numbers with a 100-step P-KL run on the default
Nemotron-Pretraining-Specialized-v1.1 (Formal-Logic) corpus, and add the
train/loss panel (now a 4-panel figure): loss 1.51->0.78, kl 2.67->0.78,
ce 0.75->0.39, accuracy 0.82->0.88; ~48k tok/s, peak 29.5 GB/GPU.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

setup() asserts policy/teacher dtensor_cfg tensor_parallel_size==1 and
context_parallel_size==1 via direct indexing, but the test fixture's
dtensor_cfg omitted those keys, causing a KeyError in
test_setup_injects_vocab_sizes_into_loss_config. Add the keys to match
real configs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>