Signed-off-by: Parth Chadha <pchadha@nvidia.com>

Comments addressed: #3, #5, NVIDIA-NeMo#7, NVIDIA-NeMo#8, NVIDIA-NeMo#9, NVIDIA-NeMo#10, NVIDIA-NeMo#11.

- Rename _load_M -> _get_sparse_projection_matrix and
  _load_dense_projection -> _get_topk_projection (later removed in
  favor of module-level cache helpers below).
- Drop unused alignment_student_spans / alignment_teacher_spans
  from the cross-tokenizer batch payload.
- Remove NRL_XTOKEN_LOSS_DUMP_DIR debug-dump side effect.
- Move Fp32SparseMM, chunk_average_log_probs, valid_chunk_mask to a
  new shared module nemo_rl/algorithms/x_token/utils.py.
- Extract projection-file parsing into utils.parse_projection_file;
  tokenalign.py and loss_functions.py both go through it.
- Move per-instance projection-matrix caches to process-local caches
  in utils.get_sparse_projection_matrix / get_topk_projection. The
  driver no longer holds large CUDA tensors; each Ray worker fills
  its own cache on first loss call.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR NVIDIA-NeMo#2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (NVIDIA-NeMo#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (NVIDIA-NeMo#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (NVIDIA-NeMo#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (NVIDIA-NeMo#13, NVIDIA-NeMo#14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (NVIDIA-NeMo#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

Comments addressed: #3, #5, NVIDIA-NeMo#7, NVIDIA-NeMo#8, NVIDIA-NeMo#9, NVIDIA-NeMo#10, NVIDIA-NeMo#11.

- Rename _load_M -> _get_sparse_projection_matrix and
  _load_dense_projection -> _get_topk_projection (later removed in
  favor of module-level cache helpers below).
- Drop unused alignment_student_spans / alignment_teacher_spans
  from the cross-tokenizer batch payload.
- Remove NRL_XTOKEN_LOSS_DUMP_DIR debug-dump side effect.
- Move Fp32SparseMM, chunk_average_log_probs, valid_chunk_mask to a
  new shared module nemo_rl/algorithms/x_token/utils.py.
- Extract projection-file parsing into utils.parse_projection_file;
  tokenalign.py and loss_functions.py both go through it.
- Move per-instance projection-matrix caches to process-local caches
  in utils.get_sparse_projection_matrix / get_topk_projection. The
  driver no longer holds large CUDA tensors; each Ray worker fills
  its own cache on first loss call.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR NVIDIA-NeMo#2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (NVIDIA-NeMo#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (NVIDIA-NeMo#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (NVIDIA-NeMo#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (NVIDIA-NeMo#13, NVIDIA-NeMo#14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (NVIDIA-NeMo#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

Comments addressed: #3, #5, #7, #8, #9, #10, #11.

- Rename _load_M -> _get_sparse_projection_matrix and
  _load_dense_projection -> _get_topk_projection (later removed in
  favor of module-level cache helpers below).
- Drop unused alignment_student_spans / alignment_teacher_spans
  from the cross-tokenizer batch payload.
- Remove NRL_XTOKEN_LOSS_DUMP_DIR debug-dump side effect.
- Move Fp32SparseMM, chunk_average_log_probs, valid_chunk_mask to a
  new shared module nemo_rl/algorithms/x_token/utils.py.
- Extract projection-file parsing into utils.parse_projection_file;
  tokenalign.py and loss_functions.py both go through it.
- Move per-instance projection-matrix caches to process-local caches
  in utils.get_sparse_projection_matrix / get_topk_projection. The
  driver no longer holds large CUDA tensors; each Ray worker fills
  its own cache on first loss call.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR #2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (#13, #14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

Comments addressed: #3, #5, #7, #8, #9, #10, #11.

- Rename _load_M -> _get_sparse_projection_matrix and
  _load_dense_projection -> _get_topk_projection (later removed in
  favor of module-level cache helpers below).
- Drop unused alignment_student_spans / alignment_teacher_spans
  from the cross-tokenizer batch payload.
- Remove NRL_XTOKEN_LOSS_DUMP_DIR debug-dump side effect.
- Move Fp32SparseMM, chunk_average_log_probs, valid_chunk_mask to a
  new shared module nemo_rl/algorithms/x_token/utils.py.
- Extract projection-file parsing into utils.parse_projection_file;
  tokenalign.py and loss_functions.py both go through it.
- Move per-instance projection-matrix caches to process-local caches
  in utils.get_sparse_projection_matrix / get_topk_projection. The
  driver no longer holds large CUDA tensors; each Ray worker fills
  its own cache on first loss call.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR #2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (#13, #14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>