Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

Signed-off-by: Yuki Huang <yukih@nvidia.com>

…t API

Address easy-batch items from PR NVIDIA-NeMo#2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (NVIDIA-NeMo#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (NVIDIA-NeMo#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (NVIDIA-NeMo#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (NVIDIA-NeMo#13, NVIDIA-NeMo#14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (NVIDIA-NeMo#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR NVIDIA-NeMo#2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (NVIDIA-NeMo#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (NVIDIA-NeMo#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (NVIDIA-NeMo#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (NVIDIA-NeMo#13, NVIDIA-NeMo#14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (NVIDIA-NeMo#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR #2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (#13, #14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>

…t API

Address easy-batch items from PR #2508 review.

- Move sinkhorn_one_dim / apply_canonicalization_if_enabled /
  clean_model_name_for_filename / project_token_likelihoods to
  nemo_rl/utils/x_token/_shared.py and import from there in the four
  projection-prep CLIs (#5).
- Drop unused sinkhorn (10-iter), debug_projection_map, and
  generate_projection_map from minimal_projection_generator.py (#8).
- Move minimal_projection_generator.py's CLI parsing under
  `if __name__ == "__main__":` so the module is importable for the
  P1 dedup harness.
- Rename seq1 / seq2 (and the derivative s1_*/s2_*/used_seq*/joined_seq*/
  seg* names) to student_tokens / teacher_tokens throughout
  TokenAligner._align_single / _align_with_anchors / _align_dp; all call
  sites are internal (#11).
- Replace **kwargs in TokenAligner._align_with_anchors with the five
  explicit keyword-only scoring knobs already declared on _align_dp,
  and pass them through named at the _align_single call site (#12).
- Remove dead TokenAligner.load_projection_matrix + the private
  _load_projection_components plus the now-orphaned self._projection_*
  attributes; the live projection-load path is
  nemo_rl/algorithms/x_token/utils.py::{get_sparse_projection_matrix,
  get_topk_projection} added in 6336464 (#13, #14).
- git mv nemo_rl/algorithms/x_token/tokenalign.py -> token_aligner.py
  and update import sites in __init__.py, data/cross_tokenizer_collate.py,
  the four utils.x_token CLIs, and the xtoken-distillation guide; update
  Sphinx :mod: references in algorithms/x_token/utils.py (#16).

Incidental: the CLIs previously called TokenAligner._canonical_token,
which was never a class attribute (the helper is module-level); the
new shared helper imports _canonical_token directly so the
use_canonicalization branch isn't broken at runtime.

Signed-off-by: Adithya Hanasoge <avenkateshha@nvidia.com>