Skip to content

feat: add fallback support for value metric type#6655

Merged
wozniakjan merged 3 commits intokedacore:mainfrom
y-rabie:support-fallback-for-value-type
Sep 12, 2025
Merged

feat: add fallback support for value metric type#6655
wozniakjan merged 3 commits intokedacore:mainfrom
y-rabie:support-fallback-for-value-type

Conversation

@y-rabie
Copy link
Contributor

@y-rabie y-rabie commented Mar 27, 2025

  • Since the constraint on having fallback only for AverageValue seems to me kinda unwarranted, it's here relaxed a bit, and can be removed altogether if we opt for another implementation.

    The constraint now becomes that the scaleTargetRef object has a field .status.readyReplicas that its controller updates with the number of ready replicas, so that we can directly use that. This is de facto the case with Deployments/StatefulSets/Replicasets/Argo Rollouts.

    We can then generically fetch the object as unstructured and access the value of the field to divide by it. A brief math illustration starting with the HPA's equation

    desiredReplicas = ceil [currentReplicas * (currentMetricValue/desiredMetricValue) ]

    By passing currentMetricValue = desiredMetricValue * fallbackReplicas / currentReplicas

    We end up with

    desiredReplicas = ceil [currentReplicas * (( desiredMetricValue * fallbackReplicas / currentReplicas )/desiredMetricValue) ]

    desiredReplicas = ceil [currentReplicas * (fallbackReplicas / currentReplicas ) ] = ceil [fallbackReplicas] = fallbackReplicas

    Emphasis: currentReplicas in HPA's equation is the number of ready replicas.

    I preferred this approach to the other one (which would remove the .status.readyReplicas field constraint) which is manually counting the number of ready pods (similar to what HPA does here), since it'd be quite involved. If it seems a better approach to you, I can implement it.

  • For full clarity, a problematic nit with this: is that we're dependent on the object's controller updating the .status.readyReplicas in a timely manner.

    If there had been a lag, then the currentReplicas the HPA multiplies by (which it gets by manually counting pods) would deviate from the currentReplicas value we divide by.

    If ours is less, then we'd scale higher than fallbackReplicas for a brief while; if ours is more, then we'd scale less than fallbackReplicas. Either way we should eventually stabilize at fallbackReplicas exactly.

  • Final unrelated small change for correctness sake (that I think should be fixed regardless of this PR), the line

    Value: *resource.NewMilliQuantity(normalisationValue*1000*replicas, resource.DecimalSI),

    has been changed to

    Value: *resource.NewMilliQuantity(int64(normalisationValue*1000*replicas), resource.DecimalSI),

    with normalizationValue being float64 instead of int.

    This prevents early casting to int, which would discard fractions in normalizationValue early on, that would have been already rounded when multiplying by 1000 below.

    Imagine normalisationValue=2.5 and fallbackReplicas=10, previously, Value = 2*1000*10 = 20000.

    Now, Value = 2.5*1000*10 = int64(25000.0) = 2500.

    Obviously the former will cause HPA to not scale to fallbackReplicas exactly.

For the unchecked list item, if this looks good, I'll open a PR to the docs repo.

Checklist

Fixes #4205

@y-rabie y-rabie requested a review from a team as a code owner March 27, 2025 03:18
@y-rabie y-rabie force-pushed the support-fallback-for-value-type branch 5 times, most recently from aeb2c2d to 3349924 Compare March 27, 2025 03:56
@stale
Copy link

stale bot commented May 26, 2025

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label May 26, 2025
@y-rabie
Copy link
Contributor Author

y-rabie commented May 28, 2025

@JorTurFer Is this something you guys would be interested in? If not, since it's stale now, I'll close it

@stale stale bot removed the stale All issues that are marked as stale due to inactivity label May 28, 2025
@JorTurFer
Copy link
Member

This is interesting indeed, the limitation of the metric type is something to fix. Sorry for not reviewing the PR, it was missed on my notification, I'm going to review it

@JorTurFer
Copy link
Member

JorTurFer commented Jun 1, 2025

/run-e2e fallback
Update: You can check the progress here

Copy link
Member

@JorTurFer JorTurFer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, the code looks nice, and it's a quite interesting solution! I'd need to check it by myself to be 100% sure that it works as expected, so let's see how e2e test goes. I've kept one small comment inline

@JorTurFer
Copy link
Member

@wozniakjan PTAL too

@SanthanCH
Copy link

@JorTurFer can we expect this in the next release?? , the version update was causing many issues as the average value is enforced , this would be helpful for me to update to latest version without any schema change for all the existing scaled objects

@rickbrouwer
Copy link
Member

I see there are also merge conflicts in this PR

@SanthanCH
Copy link

@y-rabie hey can you please take a look into it , and resolve the merge conflicts , this is much needed feature

@y-rabie y-rabie force-pushed the support-fallback-for-value-type branch 4 times, most recently from 0d0f680 to f302bd8 Compare July 15, 2025 09:05
@y-rabie
Copy link
Contributor Author

y-rabie commented Jul 15, 2025

@SanthanCH @rickbrouwer Conflicts resolved

@rickbrouwer
Copy link
Member

rickbrouwer commented Jul 15, 2025

/run-e2e fallback
Update: You can check the progress here

@SanthanCH
Copy link

@rickbrouwer can you review now , i could see all the checks passed

@rickbrouwer
Copy link
Member

@rickbrouwer can you review now , i could see all the checks passed

I think it's good to add this PR to the Release 2.18 list. That gives me some time to take a closer look at the PR, but overall, everything looks good.

@SanthanCH
Copy link

Hey @rickbrouwer is this feature going to be in next release? And when is the next release , is there any ETA ?

@y-rabie
Copy link
Contributor Author

y-rabie commented Sep 1, 2025

@JorTurFer Yep, I think the timeout was introduced because we now test rollouts and deployments. I've splitted them into two files

@y-rabie y-rabie force-pushed the support-fallback-for-value-type branch from 2a545c5 to e373fc8 Compare September 1, 2025 18:20
@SanthanCH
Copy link

Hey @JorTurFer ,@zroubalik ,@rickbrouwer
When is the next release planned ? , waiting for this feature from long time 😔🫠🙏

@rickbrouwer
Copy link
Member

rickbrouwer commented Sep 3, 2025

/run-e2e fallback*
Update: You can check the progress here

@zroubalik
Copy link
Member

@SanthanCH unfortunately e2e tests still fail, PTAl

@SanthanCH
Copy link

Hey @y-rabie could you PTAL

Signed-off-by: y-rabie <youssef.rabie@procore.com>
Signed-off-by: y-rabie <youssef.rabie@procore.com>
@y-rabie y-rabie force-pushed the support-fallback-for-value-type branch 3 times, most recently from de92e68 to 1095496 Compare September 8, 2025 00:11
Signed-off-by: y-rabie <youssef.rabie@procore.com>
@y-rabie y-rabie force-pushed the support-fallback-for-value-type branch from 1095496 to 7add249 Compare September 8, 2025 00:15
@rickbrouwer
Copy link
Member

rickbrouwer commented Sep 8, 2025

/run-e2e fallback*
Update: You can check the progress here

@SanthanCH
Copy link

Hey @JorTurFer ,@zroubalik ,@rickbrouwer
All the checks are passed now , could you please review

@rickbrouwer rickbrouwer added the ok-to-merge This PR can be merged label Sep 12, 2025
@wozniakjan wozniakjan merged commit e438e93 into kedacore:main Sep 12, 2025
25 checks passed
jmickey pushed a commit to jmickey/keda that referenced this pull request Sep 30, 2025
* feat: add fallback support for value metric type

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* Alternative implementation for getReadyReplicasCount in fallback

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* fix: restructure fallback tests to avoid timeouts

Signed-off-by: y-rabie <youssef.rabie@procore.com>

---------

Signed-off-by: y-rabie <youssef.rabie@procore.com>
@SanthanCH
Copy link

SanthanCH commented Oct 10, 2025

Still getting this error in keda admission webhook pod

ERROR    scaledobject-validation-webhook    validation error    {"name": "scaledobject", "error": "at least one trigger (that is not cpu or memory) has to have the AverageValuetype for the fallback to be enabled"}

This scaled object has only cpu trigger and metrics type is utilization

@JorTurFer
Copy link
Member

Fallback doesn't work for CPU and Memory scalers because they those metrics aren't managed by KEDA. KEDA generates the HPA for the k8s metrics server in your behalf but nothing else. To use fallback after 2.18, you need at least one scaler different from memory or cpu

@recollir
Copy link

Fallback doesn't work for CPU and Memory scalers because they those metrics aren't managed by KEDA. KEDA generates the HPA for the k8s metrics server in your behalf but nothing else. To use fallback after 2.18, you need at least one scaler different from memory or cpu

We also run into this. And did not see it in the documentation of the cpu scaler. Maybe it is stated somewhere else. Would you accept a PR for the docs about this?

@inrdkec
Copy link

inrdkec commented Oct 17, 2025

It is not in the CPU and mem scaler doc, but it is here: https://keda.sh/docs/2.18/reference/scaledobject-spec/#fallback
But it is actually a good point to improve the cpu and mem scaler doc

Fallback is supported for all triggers of both Value and AverageValue metric types, except CPU & memory triggers. It’s also only supported by ScaledObjects, not ScaledJobs.

@recollir
Copy link

@inrdkec 🙏 . Thanks for pointing out the right place in the documentation.

alt-dima pushed a commit to alt-dima/keda that referenced this pull request Dec 13, 2025
* feat: add fallback support for value metric type

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* Alternative implementation for getReadyReplicasCount in fallback

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* fix: restructure fallback tests to avoid timeouts

Signed-off-by: y-rabie <youssef.rabie@procore.com>

---------

Signed-off-by: y-rabie <youssef.rabie@procore.com>
Signed-off-by: Dmitriy Altuhov <altuhovd@gmail.com>
tangobango5 pushed a commit to tangobango5/keda that referenced this pull request Dec 22, 2025
* feat: add fallback support for value metric type

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* Alternative implementation for getReadyReplicasCount in fallback

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* fix: restructure fallback tests to avoid timeouts

Signed-off-by: y-rabie <youssef.rabie@procore.com>

---------

Signed-off-by: y-rabie <youssef.rabie@procore.com>
tangobango5 pushed a commit to tangobango5/keda that referenced this pull request Feb 13, 2026
* feat: add fallback support for value metric type

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* Alternative implementation for getReadyReplicasCount in fallback

Signed-off-by: y-rabie <youssef.rabie@procore.com>

* fix: restructure fallback tests to avoid timeouts

Signed-off-by: y-rabie <youssef.rabie@procore.com>

---------

Signed-off-by: y-rabie <youssef.rabie@procore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-merge This PR can be merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fallback for metricType: Value

10 participants