…from #7699

Cherry-picks the following from jthomson04/tokenize-endpoint:
- POST /v1/tokenize and /v1/detokenize HTTP endpoints
- Tokenizer trait: encode_with_special_tokens(), convert_ids_to_tokens()
- return_tokens_as_token_ids parameter for chat completions
- Multi-instance tokenize fix (discovery watcher)
- Jail logprobs preservation through tool-call jailing

…robs

Add return_tokens_as_token_ids support to the SGLang decode handler,
mirroring what PR #7699 added for vLLM. When enabled, logprob token
fields are returned as "token_id:<id>" instead of decoded text.

Changes:
- decode_handler.py: Read return_tokens_as_token_ids from output_options,
  pass through _process_token_stream to _extract_logprobs, format token
  strings accordingly
- sglang_processor.py: Forward return_tokens_as_token_ids through
  _build_dynamo_preproc output_options
- vllm/handlers.py: Remove debug print left in cherry-picked code

…7699)

Manually cherry-picked the /tokenize and /detokenize endpoint code from
ai-dynamo/dynamo PR #7699 (jthomson04/tokenize-endpoint). The full PR
had merge conflicts due to divergent chat_completions.rs refactoring,
so only the tokenization-specific changes were applied:

- New file: protocols/openai/tokenization.rs (request/response types)
- Tokenizer trait additions: encode_with_special_tokens, convert_ids_to_tokens
- HTTP routes: POST /tokenize, POST /detokenize
- Route wiring in service_v2.rs

These endpoints follow vLLM's /tokenize and /detokenize semantics,
enabling consistent tokenization between the Rust frontend and Python
clients for RL training.

All rl_admin functionality has been re-implemented natively in the Dynamo
Rust frontend (lib/llm):

  - Admin routes (/pause, /resume, /update_weights, /health, /ready,
    /load_lora_adapter, /unload_lora_adapter) → openai.rs /v1/rl/* router
  - Token injection (prompt_token_ids, choice.token_ids) → handler_chat_completions
  - Tokenization → /v1/tokenize + /v1/detokenize (cherry from #7699)

Prime-RL now points admin_base_url directly at the Dynamo frontend
(/v1/rl); no separate Python sidecar is deployed. The module has zero
imports across the codebase and is not referenced in any Dockerfile,
Helm chart, or K8s manifest.

Cleanup-only — no behavior change. Strips review-tracker noise that
accumulated on top of PR-added text during iteration:

  - "Closes hhzhang16 HH-19/HH-21/HH-22/HH-23/HH-25/HH-26/HH-27"
  - "CR-8 / CR-9 / CR-10 closure" prefixes on serde-error / doc-attach fixes
  - Branch-name references: bis/parity-tokenize-tcp, bis/prime-rl-merged
  - Internal PR numbers: #6094, #7699, #8197, #9141
  - Phase numbers from internal design docs (rl-support.md Phase 1/4/5)
  - "prime-rl" mentions in narrative copy and mermaid diagrams →
    generic "RL trainer / RL orchestrator / external client"

Technical content (semantics, invariants, why-this-exists rationale)
preserved everywhere; only the internal-process scaffolding is removed.

Scope verification: every removed line is one this branch ADDED
(diff main..HEAD shows the removed text on a `+` line). No edits land
on pre-existing main-branch comments. Specifically reverted the
nvext.rs cleanup attempt — its target lines (GAIE Stage 1/2,
SGLang-specific) live on main, not in this PR's diff.

Files touched:
  components/src/dynamo/vllm/handlers.py            +9  -10
  components/src/dynamo/vllm/worker_factory.py      +6  -4
  docs/dynamo-RL-api.md                             +19 -32
  lib/llm/src/http/service/openai.rs                +32 -34
  lib/llm/src/protocols/openai/chat_completions/delta.rs  +4 -4
  lib/llm/src/protocols/openai/completions/delta.rs       +3 -3
  lib/llm/src/protocols/openai/validate.rs                +20 -20

cargo check -p dynamo-llm: clean (1 pre-existing benign warning).

…e shutdowns, tier the docs

Stack-ranked review fixes from the latest Graham-style pass on this branch.
Tier 4 (commit message conventions and tokenizer-crate scope) deliberately
deferred.

Tier 1 — correctness / contract claims
  - Drop the word "atomic" from the `/v1/rl/update_weights` docblock and
    spell out the partial-failure semantics: workers `0..N-1` may have
    switched while worker `N` failed; per-worker status is in the
    response, true rollback is a follow-up.
  - Reject `n > 1` on `/v1/chat/completions` when RL token IDs are
    requested. The streaming aggregator's `completion_token_ids` Vec is
    shared across all choices, so per-choice promotion downstream cannot
    recover which tokens belong to which choice with `n > 1`. Hard-reject
    is the interim guard until the keyed-by-index refactor lands.
  - `update_weights` empty `weight_dir` (`""`) is now treated the same
    as null/missing (NCCL-mode no-op) instead of being forwarded to the
    engine as `path=""`.

Tier 2 — hard rules
  - Hot-path `[RL]` `logger.info` → `logger.debug` for pause / resume /
    flush_cache / weights-load / LoRA load / LoRA unload (8 sites). RL
    trainers fire these per training step; info-level was a log flood.
  - Extract the duplicated 4-line `EngineDeadError` shutdown stanza into
    `BaseWorkerHandler._shutdown_on_engine_dead(e) -> NoReturn` and
    collapse the 9 call sites (8 RL handlers + 1 generate path) to a
    single line each. ~32 lines removed.
  - Strip the remaining internal-tracker comments missed by an earlier
    chore-scrub: `service_v2.rs` had `jthomson04 PR #7699`,
    `bis-dev/design-docs/rl-support.md §1`, and `hhzhang16 HH-22 / HH-26`
    references in two places. Replaced with neutral prose.
  - Strip the SGLang-coordination comment in `handlers.py` ("Signatures
    intentionally line up with the SGLang RL admin routes"). The kind of
    line that goes stale when SGLang's admin set drifts.
  - Delete dead-code carcasses for the dropped routes
    (`/v1/chat/completions/tokens`, `/v1/tokenize`, `/v1/detokenize`):
    remove `tokenize`, `detokenize`, `tokenization_router`,
    `chat_completions_tokens_router`, `handler_chat_completions_tokens`
    (~240 lines), drop `pub mod tokenization;`, delete the
    `tokenization.rs` module file (124 lines), drop unused tokenize-type
    imports. All five were behind `#[allow(dead_code)]` and unmounted.

Tier 3 — tests
  - Make `RlState::new(...)` a pub(super) test-friendly constructor so
    handler-level tests don't need `from_env` / process env vars.
  - Convert `RlPauseQuery::mode: Option<String>` to `Option<PauseMode>`,
    a typed enum with `serde(rename_all = "lowercase")`. Axum now returns
    400 on `mode=foo` before the handler runs; the runtime string match
    is gone.
  - Add four behavior tests in `mod tests`:
      test_pause_mode_serde_roundtrip
      test_pause_mode_rejects_unknown_value
      test_rl_update_weights_body_defaults
      test_rl_state_new_constructs_without_env
  - Fix unrelated test struct-init breakage that shipped earlier in
    this branch: 26 sites across 11 files were missing the
    `prompt_token_ids`, `return_token_ids`, `tokens`, and
    `completion_token_ids` fields added to the response/request/nvext
    structs. `cargo test -p dynamo-llm` now compiles cleanly.

Tier 5 — design / nits
  - `DYN_RL_LIVENESS_TIMEOUT_MS` is read once in `RlState::from_env` and
    cached as `RlState.probe_timeout: Duration`; `rl_ready` and
    `rl_liveness` use the cached value instead of re-parsing the env on
    every request.
  - `rl_ready` worker-probe body simplified from a four-link
    `.ok().and_then(Result::ok).map(...).unwrap_or(false)` chain to a
    `match (timeout, send_result)` that surfaces the timeout/network
    distinction for a future log line.

cargo check -p dynamo-llm: clean (1 pre-existing benign warning).
cargo test -p dynamo-llm --lib: 58 passed (4 new RL tests).
cargo test -p dynamo-llm --tests integration suites: all green.