Summary

First Atropos environment to populate distill_token_ids / distill_logprobs on ScoredDataGroup, enabling on-policy distillation (OPD) training for agentic tasks.

Background

Based on OpenClaw-RL (Princeton, March 2026), which proved that next-state signals from agent interactions contain two forms of wasted training data:

1. Evaluative signals — implicit +1/-1 scores (test pass/fail, error traces)
2. Directive signals — token-level correction info ("you should have checked the file first")

Their combined OPD+RL method improved personalization from 0.17 → 0.81 in 36 conversations, and tool-call accuracy by 76% using process rewards vs outcome-only.

What this PR adds

environments/agentic_opd_env.py — a new Atropos environment that:

OPD Pipeline (the novel part):

• Walks the agent conversation to find (assistant_turn, next_state) pairs
• Uses an LLM judge with majority voting to extract hindsight hints from next-state signals (tool results, error messages, test verdicts)
• Builds enhanced prompts (original context + hint)
• Scores student tokens under the enhanced distribution using Atropos's built-in get_logprobs API (VLLM prompt_logprobs)
• Packages the teacher's top-K predictions as distill_token_ids / distill_logprobs on ScoredDataGroup

Task: Coding problems with test verification

• 8 built-in coding tasks (fizzbuzz, two_sum, merge_intervals, etc.)
• HuggingFace dataset support for custom coding benchmarks
• Rich next-state signals from test pass/fail, error traces, terminal output

Reward: Multi-signal scoring

• Correctness (0.7): test pass/fail via ToolContext
• Efficiency (0.15): fewer turns = better
• Tool usage (0.15): appropriate use of terminal + file tools

Configuration (AgenticOPDConfig):

• opd_enabled: toggle OPD pipeline (default: True)
• distill_topk: top-K teacher predictions per position (default: 50)
• prm_votes: majority voting count for hint judge (default: 3)
• hint_max_next_state_chars: truncation for long tool outputs (default: 4000)

WandB Metrics:

• opd/mean_hints_per_rollout, opd/mean_turns_scored, opd/hint_rate
• Standard training/eval metrics (correctness, reward, pass_rate)

Architecture

AgenticOPDEnv(HermesAgentBaseEnv)
  └── collect_trajectories() override:
        1. super().collect_trajectories() → standard agentic rollouts
        2. _apply_opd_pipeline() for each rollout:
           a. _extract_turn_pairs() — find (assistant, tool_result) pairs
           b. _extract_hint() — LLM judge with majority voting
           c. Build enhanced prompt + tokenize
           d. server.get_logprobs() — VLLM prompt_logprobs scoring
           e. Map teacher top-K back to student token positions
        3. Set distill_token_ids / distill_logprobs on ScoredDataGroup

No external servers needed — the same VLLM backend that generates rollouts also scores teacher logprobs via prompt_logprobs.

Requirements

• VLLM backend (server_type: vllm) for prompt logprob scoring
• Phase 2 mode (ManagedServer) for token-level tracking

Test Plan

• Import test passes ✅
• Hint extraction helpers verified (parse, select, inject) ✅
• Turn pair extraction logic verified ✅
• Token span search verified ✅
• Full test suite: 3376 passed (6 pre-existing failures unrelated) ✅