Skip to content

Add async tool-enabled vLLM server for GRPO training via OpenAI-compatible interface#3469

Closed
BjarniHaukur wants to merge 68 commits into
huggingface:mainfrom
ASSERT-KTH:async-vllm-server
Closed

Add async tool-enabled vLLM server for GRPO training via OpenAI-compatible interface#3469
BjarniHaukur wants to merge 68 commits into
huggingface:mainfrom
ASSERT-KTH:async-vllm-server

Conversation

@BjarniHaukur

@BjarniHaukur BjarniHaukur commented May 19, 2025

Copy link
Copy Markdown

What does this PR do?

This PR adds a new vllm_serve_async.py script to TRL. It:

  • Enables asynchronous, OpenAI-compatible inference with vLLM
  • Supports models that use tool calls (e.g., search APIs, python tool, general terminal usage)
  • Mirrors the weight syncing logic from vllm_serve.py
  • Delegates endpoint complexity to vllm.entrypoints.openai.api_server
  • Exposes a rollout_func interface that lets users define custom input/output structures and tool definitions to forward into reward functions

Fixes #3284

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a GitHub issue? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@BjarniHaukur
log answer key to wandb
a902e46
@BjarniHaukur
all Table
1fa6c69
@BjarniHaukur
HTML logging
46bf3f8
@BjarniHaukur
table
338730e
@BjarniHaukur
bump patch
c9777d3
@BjarniHaukur
hmm
d81295a
@BjarniHaukur
formatting
bd92df8
@BjarniHaukur
html esacape
05610d6
@BjarniHaukur
reward isnt string
908d739
@BjarniHaukur
sync fork
99fd8de
@BjarniHaukur
preliminary openai compatible endpoint
3943e72
@BjarniHaukur
early concept, needs refining
1723c56
@BjarniHaukur
dedupe
1491cb1
@BjarniHaukur
debug print
9a9c416
@BjarniHaukur
some slop to work on
57557da
@BjarniHaukur
unslop, missing hist
044a490
@BjarniHaukur
almost valid pseudocode
2a3c178
@BjarniHaukur
middle-ware monkey patch in mp.Pool()...
faf116c
@BjarniHaukur
remove unused
7f2c730
@BjarniHaukur
More accurate .md
1348c23
@BjarniHaukur
need gpu
fb79fb6
@BjarniHaukur
renting lambda again
b16f072
@BjarniHaukur
much nicer
14be11b
@BjarniHaukur
small
7af1273
@BjarniHaukur
aider-chat and datasets conflict
4506bad
@BjarniHaukur
risky reqs change
8388e83
@BjarniHaukur
should work, but hacky
63088e0
@BjarniHaukur
some insights, but monkeypatching probably wont suffice
8b0ed76
@BjarniHaukur
refactor: Rewrite test script to use SWE-bench dataset with MultiProc…
cea5eec 
…essAider
@BjarniHaukur
refactor: Remove logging statements from test.py
50ea732
@BjarniHaukur BjarniHaukur mentioned this pull request May 19, 2025
Open
@BjarniHaukur BjarniHaukur mentioned this pull request May 26, 2025
Closed
@tmabraham

tmabraham commented Jun 17, 2025

Copy link
Copy Markdown

thank you for this PR!!

@lewtun

lewtun commented Oct 20, 2025

Copy link
Copy Markdown
Member

Hi @BjarniHaukur thank you for the PR! We're now looking to integrate environments in TRL, so would you like to rebase your branch on main so we can test your proposal more thoroughly?

@lewtun lewtun mentioned this pull request Oct 20, 2025
Merged
9 tasks
@cmunley1 cmunley1 mentioned this pull request Nov 29, 2025
Open
@cmunley1 cmunley1 mentioned this pull request Jan 30, 2026
Merged
@qgallouedec

qgallouedec commented May 22, 2026

Copy link
Copy Markdown
Member

Hey @BjarniHaukur, thanks for opening this and for the early push toward async, OpenAI-compatible, tool-enabled rollouts.
Re-reading the PR a year on, the direction you proposed turned out to be exactly the right one, but the codebase took a longer path to get there.

Most of what this PR set out to do has landed independently, just in a different shape:

  • Async rollouts + tool calls: the experimental AsyncGRPOTrainer (trl/experimental/async_grpo/) shipped via 🔌 Asynchronous GRPO #5293 with a series of follow-ups. It includes a tool-calling loop with max_tool_calling_iterations, per-rollout tool-call tracking, and a dedicated AsyncRolloutWorker.
  • OpenAI-compatible vLLM server: I'm currently iterating on this directly inside the canonical trl vllm-serve script Make trl vllm-serve OpenAI-compatible (exploratory) #5803, rather than adding a parallel server. We probably won't merge it though, see the PR. The AsyncGRPOTrainer relies on vllm serve directly, which is openai-compatible

Going to close this PR (and #3284) as superseded, but want to be clear that "superseded" here means your proposal was correct and the rest of the project caught up to it, not that the idea was wrong. Thanks for being early on this! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support for realistic multi-step rollouts via async vLLM API

5 participants

@BjarniHaukur @tmabraham @lewtun @qgallouedec @sergiopaniego