Skip to content

[Frontend][3/N] Improve all pooling task | Support binary embedding response#27066

Merged
DarkLight1337 merged 35 commits intovllm-project:mainfrom
noooop:binary_response
Oct 22, 2025
Merged

[Frontend][3/N] Improve all pooling task | Support binary embedding response#27066
DarkLight1337 merged 35 commits intovllm-project:mainfrom
noooop:binary_response

Conversation

@noooop
Copy link
Copy Markdown
Collaborator

@noooop noooop commented Oct 17, 2025 •
edited by github-actions Bot
Loading

TL;DR

  • support endianness: ["native", "big", "little"], native by default
  • support bytes encoding_format, a very simple (but highly efficient) binary embedding response method
    • First, write metadata into the headers, then write all the binary data in order into Response body.
    • When reading, first read the headers, then read the body according to the offset into the corresponding tenser
  • This API provides three benefits @uasan
  1. Significant reduction in response size
  2. No need for JSON.parse and Base64.decode
  3. Possibility of stream processing of the response
  4. Endianness customizations

Improve all pooling task

These PRs are mostly conflicting with each other, so combining them into a series would better inform reviewers about what happened. And what else needs to be done after that?

Purpose

Response compression if the client sends the 'accept-encoding' header: 'zstd, gzip'
Use Response compression to transmit binary files that don't need base64, cool. base64 is very inefficient
I wouldn't think of this hacky method before seeing this issue.

Thanks @uasan for this cool idea

Fix #27063

cc @christian-pinto @maxdebayser @DarkLight1337

Test Plan

tests/utils_/test_serial_utils.py
tests/entrypoints/pooling/openai/test_embedding.py
tests/entrypoints/pooling/openai/test_pooling.py

Test Result

pass

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@noooop noooop changed the title [Frontend][1/N] Improve all pooling task | Support binary Embedding response by response compression [Frontend][3/N] Improve all pooling task | Support binary Embedding response by response compression Oct 17, 2025
@mergify mergify Bot added the frontend label Oct 17, 2025
@noooop noooop mentioned this pull request Oct 17, 2025
Merged
5 tasks
@noooop
init
92a6d2e 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
init
e54fe8b 
Signed-off-by: wang.yuqi <noooop@126.com>
noooop added 2 commits October 17, 2025 11:37
@noooop
+ response_compression_pooling_output
83803c4 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
fix
fc826d1 
Signed-off-by: wang.yuqi <noooop@126.com>
noooop
noooop commented Oct 17, 2025
View reviewed changes
Comment thread vllm/entrypoints/openai/utils.py Outdated
@noooop noooop changed the title [Frontend][3/N] Improve all pooling task | Support binary Embedding response by response compression [Frontend][3/N] Improve all pooling task | Support binary embedding response Oct 18, 2025
@noooop noooop changed the title [Frontend][3/N] Improve all pooling task | Support binary embedding response [Frontend][4/N] Improve all pooling task | Support binary embedding response Oct 18, 2025
This was referenced Oct 18, 2025
Merged
Merged
Merged
Merged
@noooop
support endianness
3b10e99 
Signed-off-by: wang.yuqi <noooop@126.com>
uasan
uasan reviewed Oct 19, 2025
View reviewed changes
Comment thread vllm/entrypoints/openai/protocol.py Outdated
@noooop
+ tensor_serial
fa03e6c 
Signed-off-by: wang.yuqi <noooop@126.com>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Oct 20, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @noooop.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 20, 2025
@noooop
+ embedding_requests_base64_client
b033bda 
Signed-off-by: wang.yuqi <noooop@126.com>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Oct 20, 2025

Documentation preview: https://vllm--27066.org.readthedocs.build/en/27066/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Oct 20, 2025
@noooop
Merge branch 'main' into binary_response
24bb3cf 
Signed-off-by: wang.yuqi <noooop@126.com>
@mergify mergify Bot removed the needs-rebase label Oct 20, 2025
noooop added 4 commits October 20, 2025 16:13
@noooop
fix
63502bb 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
typo
c8a6d0f 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
Support binary embedding response
f4882a4 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
+ embedding_requests_bytes_client.py
836fc05 
Signed-off-by: wang.yuqi <noooop@126.com>
maxdebayser
maxdebayser reviewed Oct 20, 2025
View reviewed changes
Comment thread vllm/utils/serial_utils.py
maxdebayser
maxdebayser reviewed Oct 20, 2025
View reviewed changes
Comment thread vllm/utils/serial_utils.py Outdated
maxdebayser
maxdebayser reviewed Oct 20, 2025
View reviewed changes
Comment thread vllm/utils/serial_utils.py Outdated
maxdebayser
maxdebayser reviewed Oct 20, 2025
View reviewed changes
Comment thread vllm/utils/serial_utils.py Outdated
noooop
noooop commented Oct 21, 2025
View reviewed changes
Comment thread vllm/utils/serial_utils.py
noooop added 5 commits October 21, 2025 08:54
@noooop
rename
3a1302b 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
StreamingResponse
07b2508 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
Merge branch 'main' into binary_response
68276e6
@noooop
rename
405f11d 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
fix
95986f1 
Signed-off-by: wang.yuqi <noooop@126.com>
@noooop
Copy link
Copy Markdown
Collaborator Author

noooop commented Oct 22, 2025

@DarkLight1337

Are there any more modifications needed for this PR?

DarkLight1337
DarkLight1337 reviewed Oct 22, 2025
View reviewed changes
Comment thread vllm/entrypoints/openai/serving_embedding.py
noooop
noooop commented Oct 22, 2025
View reviewed changes
Comment thread vllm/entrypoints/openai/api_server.py
@DarkLight1337 DarkLight1337 merged commit 1f633b8 into vllm-project:main Oct 22, 2025
51 checks passed
@noooop noooop deleted the binary_response branch October 22, 2025 10:40
@noooop noooop restored the binary_response branch October 22, 2025 11:04
@noooop noooop deleted the binary_response branch October 22, 2025 11:08
usberkeley pushed a commit to usberkeley/vllm that referenced this pull request Oct 23, 2025
@noooop @gemini-code-assist
@DarkLight1337 @usberkeley
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
be4c84b 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
@DarkLight1337 DarkLight1337 mentioned this pull request Oct 24, 2025
Merged
5 tasks
@uasan uasan mentioned this pull request Oct 25, 2025
Closed
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@noooop @gemini-code-assist
@DarkLight1337 @0xrushi
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
8a2b9ca 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
@noooop @gemini-code-assist
@DarkLight1337 @0xrushi
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
9e76f88 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
This was referenced Oct 28, 2025
Merged
Closed
ilmarkov pushed a commit to neuralmagic/vllm that referenced this pull request Nov 7, 2025
@noooop @gemini-code-assist
@DarkLight1337 @ilmarkov
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
bcbfb20 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
@noooop @gemini-code-assist
@DarkLight1337 @rtourgeman
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
235e0a9 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
@noooop @gemini-code-assist
@DarkLight1337 @devpatelio
[Frontend][3/N] Improve all pooling task | Support binary embedding r…
62debb5 
…esponse (vllm-project#27066)

Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner

+3 more reviewers

@maxdebayser maxdebayser maxdebayser left review comments

@uasan uasan uasan left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation frontend ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Improvements to front-end embedding response

4 participants

@noooop @maxdebayser @uasan @DarkLight1337