feat: Add replicas and minAvailable fields for PodCliquesScalingGroups by julienmancuso · Pull Request #116 · ai-dynamo/grove

julienmancuso · 2025-07-21T22:05:27Z

Here is an example for :

apiVersion: grove.io/v1alpha1
kind: PodGangSet
metadata:
  name: simple1
  labels:
    app: simple1
spec:
  replicas: 1
  template:
    cliques:
      - name: pca
        spec:
          roleName: rolea
          replicas: 2
          podSpec:
            containers:
              - name: pca
                image: nginx:latest
                resources:
                  requests:
                    cpu: 10m
          autoScalingConfig:
            maxReplicas: 5
            metrics:
              - type: Resource
                resource:
                  name: cpu
                  target:
                    type: Utilization
                    averageUtilization: 80
      - name: pcb
        spec:
          roleName: roleb
          replicas: 2
          podSpec:
            containers:
              - name: pcb
                image: nginx:latest
                resources:
                  requests:
                    cpu: 10m
      - name: pcc
        spec:
          roleName: rolec
          replicas: 1
          podSpec:
            containers:
              - name: pcc
                image: nginx:latest
                resources:
                  requests:
                    cpu: 10m
      - name: pcd
        spec:
          roleName: roled
          replicas: 1
          podSpec:
            containers:
              - name: pcd
                image: nginx:latest
                resources:
                  requests:
                    cpu: 10m
    podCliqueScalingGroups:
      - name: sga
        minAvailable: 2
        replicas: 4
        cliqueNames:
          - pcb
          - pcc
        scaleConfig:
          maxReplicas: 6
          metrics:
            - type: Resource
              resource:
                name: cpu
                target:
                  type: Utilization
                  averageUtilization: 80

it creates these podgangs :

base podgang :

apiVersion: scheduler.grove.io/v1alpha1
kind: PodGang
metadata:
  creationTimestamp: "2025-07-25T02:29:43Z"
  generation: 28
  labels:
    app.kubernetes.io/component: podgang
    app.kubernetes.io/managed-by: grove-operator
    app.kubernetes.io/part-of: simple1
  name: simple1-0
  namespace: jsm
  ownerReferences:
  - apiVersion: grove.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PodGangSet
    name: simple1
    uid: 70eeefa0-1c48-4639-9924-b83cc58d6ba7
  resourceVersion: "1514534"
  uid: a57617dd-70b8-4758-8b18-057626adb89c
spec:
  podgroups:
  - minReplicas: 2
    name: simple1-0-pca
    podReferences:
    - name: simple1-0-pca-mjfhc
      namespace: jsm
    - name: simple1-0-pca-r7h9h
      namespace: jsm
  - minReplicas: 2
    name: simple1-0-sga-0-pcb
    podReferences:
    - name: simple1-0-sga-0-pcb-frh4t
      namespace: jsm
    - name: simple1-0-sga-0-pcb-p44pm
      namespace: jsm
  - minReplicas: 2
    name: simple1-0-sga-1-pcb
    podReferences:
    - name: simple1-0-sga-1-pcb-qhh7p
      namespace: jsm
    - name: simple1-0-sga-1-pcb-wgpjv
      namespace: jsm
  - minReplicas: 1
    name: simple1-0-sga-0-pcc
    podReferences:
    - name: simple1-0-sga-0-pcc-n4bbk
      namespace: jsm
  - minReplicas: 1
    name: simple1-0-sga-1-pcc
    podReferences:
    - name: simple1-0-sga-1-pcc-qz8n2
      namespace: jsm
  - minReplicas: 1
    name: simple1-0-pcd
    podReferences:
    - name: simple1-0-pcd-hzcck
      namespace: jsm

podgang for replica 3 :

apiVersion: scheduler.grove.io/v1alpha1
kind: PodGang
metadata:
  creationTimestamp: "2025-07-25T02:29:42Z"
  generation: 12
  labels:
    app.kubernetes.io/component: podgang
    app.kubernetes.io/managed-by: grove-operator
    app.kubernetes.io/part-of: simple1
  name: simple1-0-sga-0
  namespace: jsm
  ownerReferences:
  - apiVersion: grove.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PodGangSet
    name: simple1
    uid: 70eeefa0-1c48-4639-9924-b83cc58d6ba7
  resourceVersion: "1514449"
  uid: 1ffc75e1-49fc-49ce-8a21-d04af7d1365f
spec:
  podgroups:
  - minReplicas: 2
    name: simple1-0-sga-2-pcb
    podReferences:
    - name: simple1-0-sga-2-pcb-n4scn
      namespace: jsm
    - name: simple1-0-sga-2-pcb-b8fcs
      namespace: jsm
  - minReplicas: 1
    name: simple1-0-sga-2-pcc
    podReferences:
    - name: simple1-0-sga-2-pcc-99q7c
      namespace: jsm

podgang for replica 4 :

apiVersion: scheduler.grove.io/v1alpha1
kind: PodGang
metadata:
  creationTimestamp: "2025-07-25T02:29:42Z"
  generation: 12
  labels:
    app.kubernetes.io/component: podgang
    app.kubernetes.io/managed-by: grove-operator
    app.kubernetes.io/part-of: simple1
  name: simple1-0-sga-1
  namespace: jsm
  ownerReferences:
  - apiVersion: grove.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PodGangSet
    name: simple1
    uid: 70eeefa0-1c48-4639-9924-b83cc58d6ba7
  resourceVersion: "1514304"
  uid: 9ab48c1e-003f-4701-a170-72b36709c51a
spec:
  podgroups:
  - minReplicas: 2
    name: simple1-0-sga-3-pcb
    podReferences:
    - name: simple1-0-sga-3-pcb-2lcwp
      namespace: jsm
    - name: simple1-0-sga-3-pcb-lkr9v
      namespace: jsm
  - minReplicas: 1
    name: simple1-0-sga-3-pcc
    podReferences:
    - name: simple1-0-sga-3-pcc-knqfx
      namespace: jsm

and follwing pods (replicas pods are in pending state while all the pods of the base podgang are not running) :

and pocliques :

Ronkahn21

I will complete the review later

Ronkahn21

I had some questions regarding the base vs individual pod gang
Do we have documentation why did we decide this flow

renormalize

Thanks for the PR @julienmancuso!

1/n review, covers the API, and the webhooks primarily.

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

renormalize

Review 2/n.

I strongly believe utils/podgang.go needs a re-look as it is very complicated right now. Many utils defined there are redundant.

Will review the syncflow.go files in the next review.

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

renormalize

3/n. This is about it from me. Sorry for the delayed reviews; the PR is very large after all! Thanks!

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

unmarshall · 2025-07-31T04:28:21Z

+	Replicas *int32 `json:"replicas,omitempty"`
+	// MinAvailable specifies the minimum number of ready replicas required for the group to be considered operational.
+	// A scaling group replica is considered "ready" when its associated PodClique has sufficient ready Pods
+	// (PodClique.Status.ReadyReplicas >= PodClique.Status.MinAvailable), where a Pod is ready when its PodReady condition is True.


Please remove this as this has changed.

unmarshall · 2025-07-31T04:30:17Z

+	// MinAvailable specifies the minimum number of ready replicas required for the group to be considered operational.
+	// A scaling group replica is considered "ready" when its associated PodClique has sufficient ready Pods
+	// (PodClique.Status.ReadyReplicas >= PodClique.Status.MinAvailable), where a Pod is ready when its PodReady condition is True.
+	// If MinAvailable is breached, it will trigger gang-termination of the podGangs.


Suggested change

// If MinAvailable is breached, it will trigger gang-termination of the podGangs.

// If MinAvailable is breached, it will be used to signal that the PodCliqueScalingGroup is no longer operating with the desired availability.

will this not trigger gang-termination after the terminationDelay?

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

renormalize

Thanks for addressing all the comments!

unmarshall

Thanks for addressing the review comments

sanjaychatterjee

Approved!

sanjaychatterjee reviewed Jul 22, 2025

View reviewed changes

Comment thread operator/internal/component/podgangset/podcliquescalinggroup/podcliquescalinggroup.go

julienmancuso force-pushed the jsm/115 branch 3 times, most recently from 593e522 to c3e5275 Compare July 25, 2025 15:24

julienmancuso marked this pull request as ready for review July 25, 2025 16:21

julienmancuso requested review from dmitsh and unmarshall as code owners July 25, 2025 16:21

unmarshall requested changes Jul 25, 2025

View reviewed changes

julienmancuso force-pushed the jsm/115 branch 4 times, most recently from 95422d6 to b7930f4 Compare July 26, 2025 03:38

julienmancuso requested review from sanjaychatterjee and unmarshall July 26, 2025 03:40

sanjaychatterjee reviewed Jul 26, 2025

View reviewed changes

Ronkahn21 reviewed Jul 27, 2025

View reviewed changes

Ronkahn21 reviewed Jul 28, 2025

View reviewed changes

Comment thread operator/internal/component/podclique/pod/syncflow.go Outdated

Comment thread operator/internal/component/utils/podgang.go Outdated

Comment thread operator/internal/errors/errors.go Outdated

julienmancuso requested review from Ronkahn21 and sanjaychatterjee July 28, 2025 16:32

renormalize suggested changes Jul 29, 2025

View reviewed changes

feat: Add replicas and minAvailable fields for PodCliquesScalingGroups

53881d0

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso force-pushed the jsm/115 branch from cf27118 to 53881d0 Compare July 29, 2025 15:28

julienmancuso added 3 commits July 29, 2025 09:38

feat: merge latest main

4895051

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

feat: updated as per review

8bcb675

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

feat: updated as per review

a351543

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso requested a review from renormalize July 29, 2025 17:33

julienmancuso added 4 commits July 29, 2025 13:42

feat: updated as per review

594def5

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

feat: updated as per review

3a17507

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

feat: updated as per review

f37ed7f

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

feat: fix ci linter/validation

5e74d73

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

sanjaychatterjee reviewed Jul 30, 2025

View reviewed changes

feat: updated as per review

16d6332

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso requested a review from sanjaychatterjee July 30, 2025 13:05

renormalize suggested changes Jul 30, 2025

View reviewed changes

feat: simplified as per Saketh's review

d214f14

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso requested a review from renormalize July 30, 2025 19:00

renormalize suggested changes Jul 31, 2025

View reviewed changes

feat: simplified as per Saketh's review

1a65ded

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso requested a review from renormalize July 31, 2025 13:55

unmarshall requested changes Jul 31, 2025

View reviewed changes

julienmancuso mentioned this pull request Jul 31, 2025

PodGang Creation Should Wait for All PodClique Resources and Their Pods to Exist #124

Closed

feat: updated as per Madhav's review

4cca628

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

julienmancuso force-pushed the jsm/115 branch from 849a5db to 4cca628 Compare July 31, 2025 21:40

julienmancuso requested review from nvrohanv and unmarshall July 31, 2025 21:59

feat: fixing validation

d79ca64

Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>

renormalize approved these changes Aug 1, 2025

View reviewed changes

unmarshall approved these changes Aug 1, 2025

View reviewed changes

sanjaychatterjee approved these changes Aug 1, 2025

View reviewed changes

renormalize merged commit 37eedaf into ai-dynamo:main Aug 1, 2025

	// If MinAvailable is breached, it will trigger gang-termination of the podGangs.
	// If MinAvailable is breached, it will be used to signal that the PodCliqueScalingGroup is no longer operating with the desired availability.

Conversation

julienmancuso commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ronkahn21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ronkahn21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

renormalize left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

renormalize left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

renormalize left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julienmancuso commented Jul 21, 2025 •

edited

Loading

renormalize left a comment •

edited

Loading

renormalize left a comment •

edited

Loading