Skip to content

Optimize schedule#1339

Merged
merrymercy merged 5 commits intomainfrom
optimize-schedule
Sep 5, 2024
Merged

Optimize schedule#1339
merrymercy merged 5 commits intomainfrom
optimize-schedule

Conversation

@hnyls2002
Copy link
Copy Markdown
Collaborator

Motivation

Modifications

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@merrymercy merrymercy merged commit ab4a83b into main Sep 5, 2024
@merrymercy merrymercy deleted the optimize-schedule branch September 5, 2024 21:30
@merrymercy merrymercy mentioned this pull request Sep 13, 2024
29 tasks
@hxer7963
Copy link
Copy Markdown
Contributor

hi, @hnyls2002 @merrymercy.

I have been exploring the source code of the PrefillAdder class and the scheduler module within ModelTpServer::get_new_prefill_batch. It seems that the implementation reserves the maximum possible output token slots based on the estimated new_token_ratio before scheduling prefill requests.

However, I am curious about the motivation behind the scheduling strategy used by PrefillAdder and how it contributes to optimizing scheduling performance.

Could you provide some insights into these aspects?

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants