server: move msg diffs tracking to HTTP thread #17740
Merged
ngxson merged 9 commits intoggml-org:masterfrom Dec 4, 2025
Merged
server: move msg diffs tracking to HTTP thread #17740ngxson merged 9 commits intoggml-org:masterfrom
ngxson merged 9 commits intoggml-org:masterfrom
Conversation
ngxson
commented
Dec 3, 2025
|
|
||
| llama_token sampled; | ||
|
|
||
| common_chat_format chat_format = COMMON_CHAT_FORMAT_CONTENT_ONLY; |
Collaborator
Author
There was a problem hiding this comment.
chat_format seems to be unused, so I removed this. no idea why it was added in the first place
Collaborator
Author
|
Server tests should be OK now, running from the mirror PR: ngxson#49 |
ngxson
commented
Dec 4, 2025
tools/server/server-context.cpp
Outdated
| states.emplace_back(task.params.oaicompat_chat_syntax); | ||
|
|
||
| tasks.push_back(std::move(task)); | ||
| states.push_back(task.params.oaicompat_chat_syntax); |
Collaborator
Author
There was a problem hiding this comment.
task is already moved at this point
ggerganov
reviewed
Dec 4, 2025
tools/server/server-context.cpp
Outdated
| // send the results | ||
| json res_json = result->to_json(); | ||
| if (result->is_error()) { | ||
| res->next = [res_this = res.get(), res_type, &should_stop, states = std::move(states)](std::string & output) mutable -> bool { |
Member
There was a problem hiding this comment.
Instead of making the lambda mutable, wouldn't it be cleaner to maintain the states in the server_res_generator instance (i.e. res)?
Collaborator
Author
There was a problem hiding this comment.
Yes that's good idea. I moved it to server_response_reader instead. update() is now called automatically whenever the result is received.
Also added a safe-guard ba2af58 to avoid someone forget to call update()
aad20a6 to
1c30c28
Compare
gabe-l-hart
added a commit
to gabe-l-hart/llama.cpp
that referenced
this pull request
Dec 4, 2025
* origin/master: server: strip content-length header on proxy (ggml-org#17734) server: move msg diffs tracking to HTTP thread (ggml-org#17740) examples : add missing code block end marker [no ci] (ggml-org#17756) common : skip model validation when --help is requested (ggml-org#17755) ggml-cpu : remove asserts always evaluating to false (ggml-org#17728) convert: use existing local chat_template if mistral-format model has one. (ggml-org#17749) cmake : simplify build info detection using standard variables (ggml-org#17423) ci : disable ggml-ci-x64-amd-* (ggml-org#17753) common: use native MultiByteToWideChar (ggml-org#17738) metal : use params per pipeline instance (ggml-org#17739) llama : fix sanity checks during quantization (ggml-org#17721) build : move _WIN32_WINNT definition to headers (ggml-org#17736) build: enable parallel builds in msbuild using MTT (ggml-org#17708) ggml-cpu: remove duplicate conditional check 'iid' (ggml-org#17650) Add a couple of file types to the text section (ggml-org#17670) convert : support latest mistral-common (fix conversion with --mistral-format) (ggml-org#17712) Use OpenAI-compatible `/v1/models` endpoint by default (ggml-org#17689) webui: Fix zero pasteLongTextToFileLen to disable conversion being overridden (ggml-org#17445)
0Marble
pushed a commit
to 0Marble/llama.cpp
that referenced
this pull request
Dec 18, 2025
* server: move msg diffs tracking to HTTP thread * wip * tool call tests ok * minor : style * cont : fix * move states to server_response_reader * add safe-guard * fix * fix 2 --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Anico2
added a commit
to Anico2/llama.cpp
that referenced
this pull request
Jan 15, 2026
* server: move msg diffs tracking to HTTP thread * wip * tool call tests ok * minor : style * cont : fix * move states to server_response_reader * add safe-guard * fix * fix 2 --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
blime4
referenced
this pull request
in blime4/llama.cpp
Feb 5, 2026
* server: move msg diffs tracking to HTTP thread * wip * tool call tests ok * minor : style * cont : fix * move states to server_response_reader * add safe-guard * fix * fix 2 --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #17726
The proposed approach: Delegate state tracking to HTTP thread
task_result_statethat holds the state of the current generation "session". This object will be owned by the HTTP threadresult.update(state); this allow the result to populate partial returns, at the same time also update thestateobject