Generate loop never exits on stop with stuck tasks

Low Severity

When stop_event is set but inflight tasks are stuck in _generate_one_turn's infinite retry loop (e.g., vLLM server permanently down), _generate_loop's outer while True never exits. It repeatedly calls asyncio.wait with a 0.1s timeout, getting empty done sets, and loops back without ever checking stop_event on that path. The finally block that cancels tasks is never reached, causing the worker thread to hang.

Additional Locations (1)

• trl/experimental/async_grpo/async_rollout_worker.py#L600-L608

This might be redundant with the queue.get(timeout=self.timeout) . The timeout will raise a queue.Empty that we correctly catch

Beautiful, definetly need this !

Nit: Maybe max_turns to be explicit ? In the future maybe we will have chatlike rollouts that don't need to call ?

I think we shouldnt raise in main loop as it can stop worker_thread silently. But there is a discussion to have here.

In the opened PR #5299 . I opted for dropping the whole group with a warning.

. Because we retry HTTP errors for each requests this is a simple sane default but as we don't want to have a missing sample from the group for training.

there is no need for a sync function here.
The while True loop is equivalent to await point. The scheduler will switch the current task is the queue is full and switch to some other coroutine

Same as previous self._groups_to_score.put_nowait(group). The async queue mechanic with await point is correct and more efficient

I don't think we need asyncio.wait_for for .get() with Timeout error. We are running in a while True loop, so the await point is the correct implementation.

Server timeout reused as queue timeout causes premature stop

Medium Severity

vllm_server_timeout is reused as the RolloutQueueDataset queue poll timeout. This conflates two separate concerns: how long to wait for the vLLM server to start and how long to wait for rollout samples during training. A user who sets a short vllm_server_timeout (because their server is already running) risks premature epoch termination whenever the queue is temporarily empty, such as during weight sync pauses.

Additional Locations (1)

• trl/experimental/async_grpo/async_grpo_trainer.py#L92-L96

Metrics collator drops keys missing from first sample

Low Severity

DataCollatorForRollout.torch_call builds the batch metrics dict using only keys from metrics_list[0]. When a batch mixes samples from tool-calling groups (which include tools/call_frequency and tools/failure_frequency keys) and non-tool-calling groups (which lack those keys), any metric key absent from the first sample is silently dropped from the entire batch, leading to inaccurate logged metrics.