Skip to content

Conversation

@mbobrovskyi
Copy link
Contributor

@mbobrovskyi mbobrovskyi commented Nov 27, 2025

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Build kueue-populator artifacts for release.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

KueuePopulator: release of the new experimental sub-project called "kueue-populator". It allows to create the default ClusterQueue, ResourceFlavor and Topology. It also creates default LocalQueues in all namespaces managed by Kueue.

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Nov 27, 2025
@netlify
Copy link

netlify bot commented Nov 27, 2025

Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit a2b78c6
🔍 Latest deploy log https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/69283a3c32c19800076a8528

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 27, 2025
@mbobrovskyi
Copy link
Contributor Author

/cc @j-skiba @mimowo

@k8s-ci-robot k8s-ci-robot requested a review from mimowo November 27, 2025 09:35
@k8s-ci-robot
Copy link
Contributor

@mbobrovskyi: GitHub didn't allow me to request PR reviews from the following users: j-skiba.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @j-skiba @mimowo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.


if [ "$HELM_CHART_PUSH" = "true" ]; then
${HELM} push "${DEST_CHART_DIR}/kueue-${chart_version}.tgz" "oci://${HELM_CHART_REPO}"
${HELM} push "${DEST_CHART_DIR}/kueue-populator-${chart_version}.tgz" "oci://${HELM_CHART_REPO}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need separate script if it is a part of Kueue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily, whatever works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial idea was to make kueue-populator as independent as possible. Also as it's experimental project we are not sure if it will remain in this repository in future and we want to keep it as a separate thing. wdyt @mimowo ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to decide now: should it be part of Kueue (tagged v0.15.0) or released separately (v0.1.0)?
If we release it under v0.15.0, we won’t be able to downgrade to v0.1.0 later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's release together

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool is tightly related to Kueue. I don’t think we need to move it to a separate repository.
We moved kjob out because it isn’t required by Kueue, but this tool directly depends on Kueue.
So I think it should stay in the Kueue repo, in a separate directory under cmd — just like kueueviz and importer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool is tightly related to Kueue. I don’t think we need to move it to a separate repository.

Sure, we don't want to move to separate repo for sure, it is going to be part of Kueue repo, just an add on project like kueueviz

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think it should stay in the Kueue repo, in a separate directory under cmd — just like kueueviz and importer.

Yes, we can move under cmd in a follow up. It was moved to experiment just because it is very early days of the project, but it belongs to cmd/ conceptually

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

/test pull-kueue-populator-test-e2e-main
let me retry to check

@j-skiba
Copy link
Contributor

j-skiba commented Nov 27, 2025

kueue dependency was removed from Chart.yaml, I think it's the cause

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

kueue dependency was removed from Chart.yaml, it think it's the cause

Good catch, so I think we should:

  1. keep the dependency because without the dependency kueue-populator cannot work
  2. longer term it would be good to test e2e using the current branch so that we can test changes both in Kueue and populator, for that we can pre-install Kueue from the branch in e2e tests

@mbobrovskyi
Copy link
Contributor Author

keep the dependency because without the dependency kueue-populator cannot work

Not sure that’s a good idea. We release kueue-populator together with Kueue, so the versions should stay the same.

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

Not sure that’s a good idea. We release kueue-populator together with Kueue, so the versions should stay the same.

Eventually yes, but for 0.15.0 I'm ok with dependency on 0.14.4. Populator should work with Kueue 0.14+ which graduated TAS to Beta.

The main thing to check is if someone has pre-installed Kueue 0.15.0, then the populator should not try to downgrade

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

The main thing to check is if someone has pre-installed Kueue 0.15.0, then the populator should not try to downgrade

@j-skiba have you had a chance to check the syntax for dependency to ensure it will accept preinstalled 0.15.0?

@j-skiba
Copy link
Contributor

j-skiba commented Nov 27, 2025

no, I haven't. I will take a look

@j-skiba
Copy link
Contributor

j-skiba commented Nov 27, 2025

seems to be working

jakubskiba@jakubskiba:~/kueue$ helm install kueue oci://us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue --version="0.15.0-rc.2-43-gcfe5b222" --create-namespace --namespace=kueue-system
Pulled: us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue:0.15.0-rc.2-43-gcfe5b222
Digest: sha256:3a50a6b26199bb608445f35d8ff00ed03f3ee2efeb6fc4c04279bee565a6a492
I1127 10:12:15.146722  918110 warnings.go:110] "Warning: unrecognized format \"int32\""
I1127 10:12:15.147411  918110 warnings.go:110] "Warning: unrecognized format \"int32\""
I1127 10:12:15.148967  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.151678  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.152881  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.152918  918110 warnings.go:110] "Warning: unrecognized format \"int32\""
I1127 10:12:15.153618  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.154553  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.164850  918110 warnings.go:110] "Warning: unrecognized format \"int32\""
I1127 10:12:15.164893  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
I1127 10:12:15.338361  918110 warnings.go:110] "Warning: unrecognized format \"int32\""
I1127 10:12:15.338398  918110 warnings.go:110] "Warning: unrecognized format \"int64\""
NAME: kueue
LAST DEPLOYED: Thu Nov 27 10:12:14 2025
NAMESPACE: kueue-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
jakubskiba@jakubskiba:~/kueue$ helm install kueue-populator oci://us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue-populator --version="0.15.0-rc.2-43-gcfe5b222" --create-namespace --namespace=kueue-system --set kueue.enabled=false --wait
Pulled: us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue-populator:0.15.0-rc.2-43-gcfe5b222
Digest: sha256:63726597882a66e7f03f516f0954908206992684eb697efbe47b352a0fbf9f13
NAME: kueue-populator
LAST DEPLOYED: Thu Nov 27 10:14:44 2025
NAMESPACE: kueue-system
STATUS: deployed
REVISION: 1
jakubskiba@jakubskiba:~/kueue$ helm list -A
NAME           	NAMESPACE   	REVISION	UPDATED                                	STATUS  	CHART                                   	APP VERSION              
kueue          	kueue-system	1       	2025-11-27 10:12:14.067625242 +0000 UTC	deployed	kueue-0.15.0-rc.2-43-gcfe5b222          	v0.15.0-rc.2-43-gcfe5b222
kueue-populator	kueue-system	1       	2025-11-27 10:14:44.383061979 +0000 UTC	deployed	kueue-populator-0.15.0-rc.2-43-gcfe5b222	v0.15.0-rc.2-43-gcfe5b222

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

seems to be working

nice, so you pre-installed 0.15, and the prepopulator dependency was ok with that

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

@j-skiba IIUC you tested with the dependency?

dependencies:
  - name: kueue
    version: "~0.14.4"
    repository: "oci://registry.k8s.io/kueue/charts"
    condition: kueue.enabled

@j-skiba
Copy link
Contributor

j-skiba commented Nov 27, 2025

nice, so you pre-installed 0.15, and the prepopulator dependency was ok with that

Correct. As long as kueue.enabled is set to false it will work that way.


@j-skiba IIUC you tested with the dependency?

Yes, I did manually and the tests run like this

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

I see so you had to add -set kueue.enabled=false. I think we may rename this param to kueue.requiredAsDependency=false or something like this

@j-skiba
Copy link
Contributor

j-skiba commented Nov 27, 2025

@mimowo I'm not 100% sure but enabled seems like some kind of standard name - https://helm.sh/docs/chart_best_practices/dependencies/#conditions-and-tags

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

@mimowo I'm not 100% sure but enabled seems like some kind of standard name - https://helm.sh/docs/chart_best_practices/dependencies/#conditions-and-tags

ok, sgtm

@mbobrovskyi mbobrovskyi force-pushed the cleanup/kueue-populator-artifacts branch from c482b5a to 4d8f407 Compare November 27, 2025 11:11
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 27, 2025
@mbobrovskyi mbobrovskyi force-pushed the cleanup/kueue-populator-artifacts branch 5 times, most recently from 8865bbb to 54346b1 Compare November 27, 2025 11:40
@mbobrovskyi mbobrovskyi force-pushed the cleanup/kueue-populator-artifacts branch from 54346b1 to a2b78c6 Compare November 27, 2025 11:47
@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

/lgtm
/approve
Leaving on hold in case @j-skiba has some comments
/hold
once it merges I will do RC3 for sanity testing before the full release

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 27, 2025
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 27, 2025
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

DetailsGit tree hash: 6c5a926753d432f6258af51af22e4cf7a1eee072

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mbobrovskyi, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 27, 2025
@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

Thank you @mbobrovskyi and @j-skiba for collaborating on the release process for the sub-project 👍

- name: kueue
version: "~0.14.4"
repository: "oci://registry.k8s.io/kueue/charts"
repository: "file://../../../../../charts/kueue"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we release that chart don't we have to replace that back to "oci://registry.k8s.io/kueue/charts"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only when releasing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mike WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this will work if someone runs it this way:

helm install kueue-populator oci://us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue-populator --version=<SOME_VERSION> --create-namespace --namespace=kueue-system --set kueue.enabled=true --wait

with kueue.enabled=true then this reference:

repository: "file://../../../../../charts/kueue"

doesn't exist I think

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or is it bundled in the package "kueue-populator" already?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the question, will this work for someone installing from LWS, or another repo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I tested it after build with command

➜  kueue git:(cleanup/kueue-populator-artifacts) ✗ make artifacts IMAGE_REGISTRY=us-central1-docker.pkg.dev/k8s-staging-images/kueue GIT_TAG=v0.15.0-rc.2

➜  kueue git:(cleanup/kueue-populator-artifacts) ✗ helm install kueue-populator ./artifacts/kueue-populator-0.15.0-rc.2.tgz \
  --namespace kueue-system \
  --create-namespace \
  --set kueue.enabled=true \
  --wait --timeout 300s

➜  kueue git:(cleanup/kueue-populator-artifacts) ✗ k get pods -A                                                                        
NAMESPACE            NAME                                                  READY   STATUS             RESTARTS   AGE
kube-system          coredns-66bc5c9577-4s4tx                              1/1     Running            0          70s
kube-system          coredns-66bc5c9577-jwxfn                              1/1     Running            0          70s
kube-system          etcd-kind-control-plane                               1/1     Running            0          78s
kube-system          kindnet-dk57m                                         1/1     Running            0          71s
kube-system          kube-apiserver-kind-control-plane                     1/1     Running            0          77s
kube-system          kube-controller-manager-kind-control-plane            1/1     Running            0          77s
kube-system          kube-proxy-v75c2                                      1/1     Running            0          71s
kube-system          kube-scheduler-kind-control-plane                     1/1     Running            0          77s
kueue-system         kueue-populator-controller-manager-5f4d455cc8-dz5md   1/1     Running            0          70s
kueue-system         kueue-populator-kueue-populator-fb764bfd4-lxwhk       0/1     ImagePullBackOff   0          70s
local-path-storage   local-path-provisioner-7b8c8ddbd6-kjd8n               1/1     Running            0          69s

kueue-populator-kueue-populator-fb764bfd4-vct6m with error because we don't have image yet.
Butk kueue-populator-controller-manager-5f4d455cc8-rxp5x running successfully .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works, I tested.

If that works then it's ok for me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still thinking that we should install it separately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait we have enabled: false so we install seperately by default, only if requested by user it is combined.

It makes sense I think. In package managers like apt you usually get depedencies installed, unless you already have it installed. In any case users have a choice now.

@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

/unhold
/lgtm

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 27, 2025
@mimowo
Copy link
Contributor

mimowo commented Nov 27, 2025

Let's merge and see how RC3 is going

@k8s-ci-robot k8s-ci-robot merged commit 80bd229 into kubernetes-sigs:main Nov 27, 2025
29 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v0.15 milestone Nov 27, 2025
@mbobrovskyi mbobrovskyi deleted the cleanup/kueue-populator-artifacts branch November 27, 2025 13:21
@mbobrovskyi
Copy link
Contributor Author

Trying to test

➜  kueue git:(fix/post-kueue-push-images) ✗ helm install kueue-populator oci://us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue-populator \
  --version 0.15.0-rc.2-50-g1a36086f \
  --namespace kueue-system \
  --create-namespace \
  --wait

➜  kueue git:(fix/post-kueue-push-images) ✗ k get pods -A
NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
kube-system          coredns-66bc5c9577-cbl29                           1/1     Running   0          58s
kube-system          coredns-66bc5c9577-klvjl                           1/1     Running   0          59s
kube-system          etcd-kind-control-plane                            1/1     Running   0          66s
kube-system          kindnet-hl9gr                                      1/1     Running   0          59s
kube-system          kube-apiserver-kind-control-plane                  1/1     Running   0          65s
kube-system          kube-controller-manager-kind-control-plane         1/1     Running   0          65s
kube-system          kube-proxy-kfknb                                   1/1     Running   0          59s
kube-system          kube-scheduler-kind-control-plane                  1/1     Running   0          65s
kueue-system         kueue-populator-create-kueue-resources-job-5pfxl   0/1     Error     0          9s
kueue-system         kueue-populator-create-kueue-resources-job-ttndr   0/1     Error     0          24s
kueue-system         kueue-populator-kueue-populator-664b9ddd95-qc4z6   1/1     Running   0          56s
local-path-storage   local-path-provisioner-7b8c8ddbd6-t4t45            1/1     Running   0          59s

➜  kueue git:(fix/post-kueue-push-images) ✗ k logs -f -n kueue-system         kueue-populator-create-kueue-resources-job-5pfxl
resource mapping not found for name: "tas-gpu-default" namespace: "" from "/etc/config/resources.yaml": no matches for kind "ResourceFlavor" in version "kueue.x-k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "cluster-queue" namespace: "" from "/etc/config/resources.yaml": no matches for kind "ClusterQueue" in version "kueue.x-k8s.io/v1beta1"
ensure CRDs are installed first

@mbobrovskyi
Copy link
Contributor Author

mbobrovskyi commented Nov 27, 2025

Ah, my bad. Looks like working fine.

➜  kueue git:(fix/post-kueue-push-images) ✗ helm install kueue-populator oci://us-central1-docker.pkg.dev/k8s-staging-images/kueue/charts/kueue-populator \
  --version 0.15.0-rc.2-50-g1a36086f \
  --namespace kueue-system \
  --set kueue.enabled=true \
  --create-namespace \
  --wait

➜  kueue git:(fix/post-kueue-push-images) ✗ k get pods -A
NAMESPACE            NAME                                                READY   STATUS    RESTARTS   AGE
kube-system          coredns-66bc5c9577-hqtkd                            1/1     Running   0          65s
kube-system          coredns-66bc5c9577-ltt8n                            1/1     Running   0          65s
kube-system          etcd-kind-control-plane                             1/1     Running   0          72s
kube-system          kindnet-l4qrn                                       1/1     Running   0          65s
kube-system          kube-apiserver-kind-control-plane                   1/1     Running   0          71s
kube-system          kube-controller-manager-kind-control-plane          1/1     Running   0          71s
kube-system          kube-proxy-7gf58                                    1/1     Running   0          65s
kube-system          kube-scheduler-kind-control-plane                   1/1     Running   0          72s
kueue-system         kueue-populator-controller-manager-ffd9497f-26q2m   1/1     Running   0          50s
kueue-system         kueue-populator-kueue-populator-664b9ddd95-5kd6d    1/1     Running   0          50s
local-path-storage   local-path-provisioner-7b8c8ddbd6-cwds4             1/1     Running   0          65s

@mimowo
Copy link
Contributor

mimowo commented Nov 28, 2025

/remove-kind cleanup
/kind feature
/release-note-edit

KueuePopulator: release of the new experimental opt-in sub-project called "kueue-populator". 
It allows to create the default ClusterQueue, ResourceFlavor and Topology. It also creates default LocalQueues in all namespaces managed by Kueue.

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. release-note-none Denotes a PR that doesn't merit a release note. labels Nov 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants