Skip to content

update kops scalability run-test.sh to allow configuring qps and burst#17742

Merged
k8s-ci-robot merged 1 commit intokubernetes:masterfrom
alaypatel07:dra-5k-qps-config
Nov 5, 2025
Merged

update kops scalability run-test.sh to allow configuring qps and burst#17742
k8s-ci-robot merged 1 commit intokubernetes:masterfrom
alaypatel07:dra-5k-qps-config

Conversation

@alaypatel07
Copy link
Copy Markdown
Contributor

@alaypatel07 alaypatel07 commented Nov 5, 2025

The 5k node dra tests are not meeting thresholds because scheduler and controller-manager client side QPS is throttling processing of churn pods:

$  cat  1985905032616218624/artifacts/control-plane-us-east1-b-b99x/kube-scheduler.log | grep client-side
...
...
I1105 05:40:55.762725      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-2/small-928-5-2n8mc" delay="1m10.098265776s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/small-928-5-2n8mc/status"
I1105 05:40:56.782941      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-2/small-1205-5-jv76x" delay="1m9.833553754s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/small-1205-5-jv76x/status"
I1105 05:40:57.783140      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-2/small-244-6-99vxp" delay="1m9.31645658s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/small-244-6-99vxp/status"
I1105 05:40:58.802643      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-1/small-1000-9-4tnbf" delay="1m9.035718483s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-1/pods/small-1000-9-4tnbf/status"
I1105 05:40:59.822295      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-2/small-1222-5-2dgw8" delay="1m8.60940739s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/small-1222-5-2dgw8/status"
I1105 05:41:00.822782      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-2/small-1856-4-j7xmn" delay="1m8.276384139s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/small-1856-4-j7xmn/status"
I1105 05:41:01.842689      12 request.go:752] "Waited before sending request" pod="test-w4ft5o-1/small-1322-8-wz42f" delay="1m8.076719043s" reason="client-side throttling, not priority and fairness" verb="PATCH" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-1/pods/small-1322-8-wz42f/status"
...
...

$  cat 1985905032616218624/artifacts/control-plane-us-east1-b-b99x/kube-controller-manager.log | grep client-side
I1105 05:36:36.005792      12 request.go:752] "Waited before sending request" delay="9.000194347s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-15321-0-4pqxh"
I1105 05:36:37.008229      12 request.go:752] "Waited before sending request" delay="10.001241974s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-15786-0-84cd8"
I1105 05:36:38.009198      12 request.go:752] "Waited before sending request" delay="11.000902014s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-16479-0-ff7wx"
I1105 05:36:39.010780      12 request.go:752] "Waited before sending request" delay="12.00106924s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-15131-0-9zdg9"
I1105 05:36:40.012557      12 request.go:752] "Waited before sending request" delay="12.998646168s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-14178-0-bcx5z"
I1105 05:36:41.014169      12 request.go:752] "Waited before sending request" delay="13.9962219s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-1554-0-qnjq6"
I1105 05:36:42.014242      12 request.go:752] "Waited before sending request" delay="14.992518756s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-2/pods/long-running-14553-0-hwlv6"
I1105 05:36:43.014294      12 request.go:752] "Waited before sending request" delay="15.98861069s" reason="client-side throttling, not priority and fairness" verb="DELETE" URL="https://127.0.0.1/api/v1/namespaces/test-w4ft5o-1/pods/long-running-15289-0-dghgq"

ref: https://gcsweb.k8s.io/gcs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kops-gce-5000-node-dra-with-workload-ipalias-using-cl2/1985905032616218624/artifacts/control-plane-us-east1-b-b99x/

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 5, 2025
@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 5, 2025
@alaypatel07
Copy link
Copy Markdown
Contributor Author

alaypatel07 commented Nov 5, 2025

/assign @upodroid

@alaypatel07
Copy link
Copy Markdown
Contributor Author

alaypatel07 commented Nov 5, 2025

cc @dims

@hakuna-matatah
Copy link
Copy Markdown
Contributor

hakuna-matatah commented Nov 5, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2025
@hakman
Copy link
Copy Markdown
Member

hakman commented Nov 5, 2025

/approve

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Nov 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 5, 2025
@alaypatel07
Copy link
Copy Markdown
Contributor Author

alaypatel07 commented Nov 5, 2025

/hold

I think I will also need the max in flight env var

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 5, 2025
@alaypatel07
update kops scalability run-test.sh to allow configuring qps and burs…
e87a14d 
…t and apiserver max in flight requests

Signed-off-by: Alay Patel <alayp@nvidia.com>
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2025
@alaypatel07
Copy link
Copy Markdown
Contributor Author

alaypatel07 commented Nov 5, 2025

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 5, 2025
@alaypatel07
Copy link
Copy Markdown
Contributor Author

alaypatel07 commented Nov 5, 2025

@hakuna-matatah can you PTAL again and tag if this looks good?

@alaypatel07 alaypatel07 mentioned this pull request Nov 5, 2025
Merged
@hakuna-matatah
Copy link
Copy Markdown
Contributor

hakuna-matatah commented Nov 5, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2025
@k8s-ci-robot k8s-ci-robot merged commit 9ad7b96 into kubernetes:master Nov 5, 2025
27 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.35 milestone Nov 5, 2025
@alaypatel07 alaypatel07 deleted the dra-5k-qps-config branch November 5, 2025 19:56
This was referenced Nov 8, 2025
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hakman hakman Awaiting requested review from hakman

@hakuna-matatah hakuna-matatah Awaiting requested review from hakuna-matatah

Assignees

@upodroid upodroid

@hakuna-matatah hakuna-matatah

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Milestone

v1.35

Development

Successfully merging this pull request may close these issues.

5 participants

@alaypatel07 @hakuna-matatah @hakman @k8s-ci-robot @upodroid