What

ckpt-resume and frozen-resume functional tests previously wrote tensorboard events from both training phases into the same $TENSORBOARD_PATH directory and relied on files[1] (mtime-sorted glob index=1) to identify the resume run's data. That assumption is racy:

• if only one events file lands in the dir → IndexError (the failure mode in 🐛 CI failure(nightlies and MR runs): ckpt-resume tests on dgx_gb200 fail with IndexError when reading 2nd-run tensorboard events #4903),
• if the indexed file happens to have no scalar events flushed yet, or is some unrelated extra file → read_tb_logs_as_list returns {}, the _2nd.json file is written as {}, and the resume pytest fails in fixture setup with a pydantic ValidationError: input_value=PydanticUndefined out of read_golden_values_from_json at tests/functional_tests/python_test_utils/common.py:168 — the latest h100 incarnation observed on the gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances test case (same test enumerated in 🐛 CI failure(nightlies and MR runs): ckpt-resume tests on dgx_gb200 fail with IndexError when reading 2nd-run tensorboard events #4903).

This PR removes the glob-ordering dependency by giving each phase its own directory.

How

run_ci_test.sh

• Snapshot the per-repeat tensorboard base into _REPEAT_TENSORBOARD_PATH.
• For ckpt-resume / frozen-resume, before each training phase set:

  export TENSORBOARD_PATH="$_REPEAT_TENSORBOARD_PATH/run_${RUN_NUMBER}"
  mkdir -p "$TENSORBOARD_PATH"

  Training picks up the new path via the existing --tensorboard-dir: ${TENSORBOARD_PATH} in the recipe YAML.
• First-run analyzer now points at ${_REPEAT_TENSORBOARD_PATH}/run_1 for resume tests (unchanged for everything else).
• Second-run analyzer points at ${_REPEAT_TENSORBOARD_PATH}/run_2 and no longer passes --is-second-run.

get_test_results_from_tensorboard_logs.py

• The --is-second-run flag and the index=1 branch it controlled are removed. The single caller now passes an explicit per-phase --logs-dir, so the script always reads files[0].

Layout, before / after

# before (both phases collide; resume analyzer guesses via mtime sort)
$TENSORBOARD_PATH/
  events.out.tfevents.<ts1>.<host>.<pid1>    ← run 1
  events.out.tfevents.<ts2>.<host>.<pid2>    ← run 2 (analyzer: files[1])

# after (per-phase isolation; analyzer reads explicit subdir)
$TENSORBOARD_PATH/
  run_1/events.out.tfevents.<...>
  run_2/events.out.tfevents.<...>

Scope

• Behavior unchanged for frozen-start, release, checkpoint-consistency, inference, and RL modes — they don't write to per-run subdirs.
• No change to golden values; same scalars, same metric filter list.

Related

• Same failure family as the closed 🐛 CI failure(nightlies and MR runs): ckpt-resume tests on dgx_gb200 fail with IndexError when reading 2nd-run tensorboard events #4903 (gb200 IndexError variant). The multi-node sync fix in fix: Fix multi-node functional test phase sync #4924 didn't address single-node h100 because the underlying cause is the analyzer's index-into-sorted-glob, not multi-node phase ordering.