Summary

Reintroduces a compute_scale config knob so the benchmark can run efficiently on GPUs larger than the H100 the tasks are tuned on (e.g. H200 ≈ 2× H100), in response to #4. Set compute_scale: 0.5 for H200; the default 1.0 (H100) is a strict no-op.

The macro-batch of every adapted task is held constant, so results are unchanged — only the GPU count / per-GPU batch / wall-clock differ. Both task families are adapted through the same per-entry h200 block of shape {compute, env} — no script switching.

Design

LLM pretraining (11 llm-pretrain-* tasks)

At 0.5 the training command drops from 4 → 2 GPUs and compensates per GPU: BATCH_SIZE ×2, GRAD_ACCUM ÷2.

The macro-batch = BATCH_SIZE × GRAD_ACCUM is invariant (the script divides grad-accum by world size internally) — e.g. llm-pretrain-loss: H100 32×16=512 equals H200 64×8=512. The larger per-GPU micro-batch uses the H200's extra memory; the gradient is mathematically identical, so the metric is unchanged.

"h200": { "compute": 2.0, "env": { "BATCH_SIZE": "64", "GRAD_ACCUM": "8" } }

LLM RL (4 llm-rl-* tasks)

At 0.5 the command stays the same train.sh but drops from 2 → 1 GPU and applies its H200 profile via env (same mechanism as pretrain — no separate script):

• TP_SIZE=1 — vLLM tensor_model_parallel_size 2 → 1
• MAX_TOKEN_LEN_PER_GPU=20480 — per-GPU dynamic-batch token budget (17408 → 20480)
• GPU_MEM_UTIL=0.5 — gpu_memory_utilization (0.4 → 0.5)

train.sh is parametrised with the H100 values as defaults; the old train_1gpu.sh script-switch variants are removed. The global data.train_batch_size (128 prompts) is unchanged, so the RL result is unchanged.

"h200": { "compute": 1, "env": { "TP_SIZE": "1", "MAX_TOKEN_LEN_PER_GPU": "20480", "GPU_MEM_UTIL": "0.5" } }

Other tasks (optional)

compute_scale is offered for every task but is only really needed for the two families above:

• fractional / single-GPU entries (compute ≤ 1, e.g. eval jobs and the RL-control tasks): compute ×= scale for denser packing — never changes a single job's result.
• multi-GPU without an h200 block (compute > 1, e.g. cv-vae-loss): left untouched plus a stderr warning, since cutting a data-parallel job's GPU count alone would change its global batch / result.

Also: fix llm-rl-advantage training data

While unifying the RL scripts, found a data drift. llm-rl-advantage is a 200-step task, but its train.sh pointed at deepmath/train_5k (a 5K random subset), whereas its real single-GPU baseline — the leaderboard was produced with train_1gpu.sh on H200 — and the other two 200-step tasks (kl-estimator, reward-normalization) use deepmath/train_lv3-5 (30K). Corrected train.sh to train_lv3-5. (The 100-step importance-sampling correctly keeps train_5k.)

Implementation

• scale_test_cmd_entries() (agent/tools.py): single source of truth for the transform (deep-copies entries, strips the h200 helper key).
• test_cmd entries now honor a per-entry env dict, injected last in all execution backends: apptainer, docker, local, SLURM (via _build_container_cmd), and the rootless-docker long-lived session.
• the standalone GPU scheduler mirrors the compute-number transform so its bin-packing matches actual demand.
• base.py / cli.py / discover_agent.py forward compute_scale from config; configs/react.yaml documents the knob.

Not covered

The Harbor export path (harbor/ + datasets/.../score_task.py) runs tasks independently of the native runtime, so it does not consume compute_scale.

Testing

Static checks: entry resolution (h200 override for both families; fractional scaling; multi-GPU untouched; 1.0 no-op), per-entry env rendered into apptainer/docker commands, tools/scheduler transform parity, parametrised train.sh passes bash -n, and all task configs pass edit-range validation.

Refs #4