Skip to content

[Task] Add priority scheduling CIs for preemption path #13470

@harrisonlimh

Description

@harrisonlimh

Checklist

Motivation

There is a number token and request level memory accounting issues reported and fixed by the community recently. It would be good to complement them with robust CI tests.

Context:

  1. bugfix[schedule]: Excessive preemption occurs when preempting running requests to schedule new prefill requests. #12494
  2. Fix wrong running_bs in priority scheduling #13142
  3. Fix bug: Incorrect variable used in rem_total_token_offset calculatio… #13201; [Bug] Incorrect variable used in rem_total_token_offset calculation during preemption (line 700) #13111 (TestRemTotalTokenOffsetBug)
  4. valid_running_reqs = (
    r for r in self.running_batch.reqs if r not in self.preempt_list
    )
    - cover this case

Related resources

No response

Metadata

Metadata

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions