Skip to content

[KEP-3521] Part 2: Core scheduling implementation#113275

Merged
k8s-ci-robot merged 2 commits intokubernetes:masterfrom
Huang-Wei:kep-3521-B
Nov 8, 2022
Merged

[KEP-3521] Part 2: Core scheduling implementation#113275
k8s-ci-robot merged 2 commits intokubernetes:masterfrom
Huang-Wei:kep-3521-B

Conversation

@Huang-Wei
Copy link
Copy Markdown
Member

@Huang-Wei Huang-Wei commented Oct 22, 2022 •
edited
Loading

What type of PR is this?

/kind feature
/sig scheduling

What this PR does / why we need it:

This PR is rebased atop Part 1 (#113274), covering the following logic:

  • API, Codegen, UT for Enqueue extension point
  • Add EnqueuePlugin to Scheduler Framework
  • Implement EnqueuePluign in scheduler queue
  • Implementation of DefaultEnqueue plugin
  • Metrics

Which issue(s) this PR fixes:

Part of #113269

Special notes for your reviewer:

Does this PR introduce a user-facing change?

A new `preEnqueue` extension point is added to scheduler's component config v1beta2/v1beta3/v1.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3521-pod-scheduling-readiness

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 22, 2022
@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Oct 22, 2022

cc @ahg-g this is the 2nd PR of the overall KEP impl. There is a third PR upcoming to cover integration/e2e tests.

@k8s-ci-robot k8s-ci-robot added area/code-generation area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Oct 22, 2022
@k8s-triage-robot
Copy link
Copy Markdown

k8s-triage-robot commented Oct 22, 2022

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@Huang-Wei Huang-Wei force-pushed the kep-3521-B branch 2 times, most recently from 7dbe124 to 4de7b6b Compare October 23, 2022 18:04
@k8s-ci-robot k8s-ci-robot added area/stable-metrics Issues or PRs involving stable metrics sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Oct 23, 2022
@k8s-triage-robot
Copy link
Copy Markdown

k8s-triage-robot commented Oct 23, 2022

This PR may require stable metrics review.

Stable metrics are guaranteed to not change. Please review the documentation for the requirements and lifecycle of stable metrics and ensure that your metrics meet these guidelines.

@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Oct 24, 2022

/retest

1 similar comment
@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Oct 27, 2022

/retest

ahg-g
ahg-g reviewed Oct 27, 2022
View reviewed changes
Comment thread pkg/scheduler/internal/queue/scheduling_queue.go Outdated
Comment thread pkg/scheduler/internal/queue/scheduling_queue.go Outdated
Copy link
Copy Markdown
Member

@ahg-g ahg-g Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ErrorS instead?

Copy link
Copy Markdown
Member

@ahg-g ahg-g Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, should we log only on Error status?

Copy link
Copy Markdown
Member Author

@Huang-Wei Huang-Wei Oct 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do klog.V(5).InfoS on failures and klog.ErrorS on unexpected errors, which is consistent with what we did in other extension point:

kubernetes/pkg/scheduler/schedule_one.go

Lines 145 to 150 in a0b69ec

if status.Code() == framework.Error {
klog.ErrorS(nil, "Status after running PostFilter plugins for pod", "pod", klog.KObj(pod), "status", status)
} else {
fitError.Diagnosis.PostFilterMsg = status.Message()
klog.V(5).InfoS("Status after running PostFilter plugins for pod", "pod", klog.KObj(pod), "status", status)
}

Comment thread pkg/scheduler/framework/plugins/defaultenqueue/default_enqueue.go Outdated
Comment thread pkg/scheduler/framework/types.go Outdated
Comment thread pkg/scheduler/metrics/metrics.go Outdated
Comment thread pkg/scheduler/internal/queue/scheduling_queue.go Outdated
Comment thread pkg/scheduler/internal/queue/scheduling_queue.go Outdated
@leilajal
Copy link
Copy Markdown
Contributor

leilajal commented Oct 27, 2022

/remove-sig api-machinery

@k8s-ci-robot k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Oct 27, 2022
@k8s-ci-robot k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Oct 29, 2022
@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 7, 2022
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Nov 7, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, liggitt, logicalhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 7, 2022
@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Nov 7, 2022

@ahg-g almost all comments have been addressed. Let me know if it's good to squash.

@ahg-g
Copy link
Copy Markdown
Member

ahg-g commented Nov 7, 2022

@ahg-g almost all comments have been addressed. Let me know if it's good to squash.

looks good, please squash

@Huang-Wei
API, Codegen, UT for PreEnqueue extension point
2de75d9
@Huang-Wei
PreEnqueue implementation
0b27f25 
- Add PreEnqueuePlugin to Scheduler Framework
- Implement PreEnqueuePlugin in scheduler queue
- Implementation of SchedulingGates plugin
- Metrics
@Huang-Wei Huang-Wei mentioned this pull request Nov 7, 2022
Open
@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Nov 7, 2022

/retest

@Huang-Wei
Copy link
Copy Markdown
Member Author

Huang-Wei commented Nov 8, 2022

@ahg-g commits squashed and CI is green now.

@ahg-g
Copy link
Copy Markdown
Member

ahg-g commented Nov 8, 2022

/hold cancel
/lgtm

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 8, 2022
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 8, 2022
@binacs
Copy link
Copy Markdown
Member

binacs commented Nov 8, 2022

/retest

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Nov 8, 2022 •
edited
Loading

@Huang-Wei: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-local-e2e 4de7b6b790bb4bcb1fbab1f83bce2c73dc68bb9b link false /test pull-kubernetes-local-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@pacoxu
Copy link
Copy Markdown
Member

pacoxu commented Nov 8, 2022

/retest

@Huang-Wei Huang-Wei mentioned this pull request Nov 8, 2022
Closed
@k8s-ci-robot k8s-ci-robot merged commit 95bd687 into kubernetes:master Nov 8, 2022
@Huang-Wei Huang-Wei deleted the kep-3521-B branch November 8, 2022 07:40
pohly added a commit to pohly/kubernetes that referenced this pull request Nov 8, 2022
@pohly
scheduler: config feature gate check for dynamic resources plugin
2e18e38 
As discussed for other plugins, a plugin that only does some work when a
certain feature gate is enabled should only get included in the default
configuration when that feature gate is enabled.

The plugin still needs to check the feature gate itself because it might be
enabled explicitly in some configuration, with the feature gate off.

Ref:
- kubernetes#113705
- kubernetes#113275 (comment)
pohly added a commit to pohly/kubernetes that referenced this pull request Nov 9, 2022
@pohly
scheduler: config feature gate check for dynamic resources plugin
0d5fda5 
As discussed for other plugins, a plugin that only does some work when a
certain feature gate is enabled should only get included in the default
configuration when that feature gate is enabled.

The plugin still needs to check the feature gate itself because it might be
enabled explicitly in some configuration, with the feature gate off.

Ref:
- kubernetes#113705
- kubernetes#113275 (comment)
pohly added a commit to pohly/kubernetes that referenced this pull request Nov 9, 2022
@pohly
scheduler: config feature gate check for dynamic resources plugin
8eff6a5 
As discussed for other plugins, a plugin that only does some work when a
certain feature gate is enabled should only get included in the default
configuration when that feature gate is enabled.

The plugin still needs to check the feature gate itself because it might be
enabled explicitly in some configuration, with the feature gate off.

Ref:
- kubernetes#113705
- kubernetes#113275 (comment)
pohly added a commit to pohly/kubernetes that referenced this pull request Nov 10, 2022
@pohly
scheduler: config feature gate check for dynamic resources plugin
345e194 
As discussed for other plugins, a plugin that only does some work when a
certain feature gate is enabled should only get included in the default
configuration when that feature gate is enabled.

The plugin still needs to check the feature gate itself because it might be
enabled explicitly in some configuration, with the feature gate off.

Ref:
- kubernetes#113705
- kubernetes#113275 (comment)
@lianghao208 lianghao208 mentioned this pull request Feb 7, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liggitt liggitt liggitt left review comments

@andrewsykim andrewsykim Awaiting requested review from andrewsykim

@caesarxuchao caesarxuchao Awaiting requested review from caesarxuchao

+5 more reviewers

@denkensk denkensk denkensk left review comments

@kerthcet kerthcet kerthcet left review comments

@ahg-g ahg-g ahg-g left review comments

@sanposhiho sanposhiho sanposhiho left review comments

@logicalhan logicalhan logicalhan approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@logicalhan logicalhan

@liggitt liggitt

@ahg-g ahg-g

Labels

api-review Categorizes an issue or PR as actively needing an API review. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/code-generation area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/stable-metrics Issues or PRs involving stable metrics area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Projects

API Reviews
Status: API review completed, 1.26
[sig-release] Bug Triage
Archived in project

Milestone

v1.26

Development

Successfully merging this pull request may close these issues.