fix: restore HPA behavior when paused-scale-in/out annotation is deleted#7291
Conversation
|
Thank you for your contribution! 🙏 Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected. While you are waiting, make sure to:
Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient. Learn more about our contribution guide. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
9682ec2 to
39b0296
Compare
When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com>
39b0296 to
fb3ff54
Compare
|
/run-e2e internal |
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where removing the paused-scale-in or paused-scale-out annotation from a ScaledObject fails to restore the HPA's default scaling behavior when the ScaledObject doesn't explicitly define a selectPolicy in its behavior configuration.
Key Changes
- Added explicit
DeepEqualcheck for HPABehaviorfield to detect nil-to-value changes thatDeepDerivativemisses - Added comprehensive unit test verifying HPA behavior restoration when annotation is removed without custom ScaleDown configuration
- Added e2e test validating end-to-end scaling behavior after annotation removal
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
controllers/keda/hpa.go |
Added DeepEqual check for Behavior field before DeepDerivative comparison to properly detect when paused annotations are removed and Behavior should change from Disabled back to nil |
controllers/keda/scaledobject_controller_test.go |
Added unit test that verifies HPA's ScaleDown policy is no longer Disabled after removing paused-scale-in annotation when no custom ScaleDown behavior is defined, while preserving custom ScaleUp configuration |
tests/internals/pause_scale_in_restore/pause_scale_in_restore_test.go |
Added e2e test that validates deployment scales down correctly after removing paused-scale-in annotation |
CHANGELOG.md |
Added entry documenting the fix with PR reference |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // assert that the deployment scales down to 0 after removing the annotation | ||
| assert.True(t, WaitForDeploymentReplicaReadyCount(t, kc, deploymentName, testNamespace, 0, 60, 2), | ||
| "replica count should be 0 after removing paused-scale-in annotation") | ||
|
|
There was a problem hiding this comment.
This e2e test verifies the end-to-end behavior (that scaling works after removing the annotation), but doesn't directly verify that the HPA's ScaleDown.SelectPolicy is no longer Disabled. Consider adding an assertion to check the HPA spec directly, similar to what the unit test does in scaledobject_controller_test.go, to make the test more explicit about what it's verifying.
| It("removes scale down disabled policy when annotation removed without custom scaledown behavior", func() { | ||
| // This test verifies that when paused-scale-in annotation is deleted | ||
| // and no custom ScaleDown behavior is defined in the ScaledObject spec, but ScaleUp is. | ||
| // The HPA's ScaleDown should be cleared (nil) instead of remaining as Disabled, | ||
| // while ScaleUp should be preserved. | ||
| // NOTE: We include another annotation to prevent the edge case where removing the | ||
| // only annotation makes annotations nil, which triggers update via annotation check. | ||
| deploymentName := "remove-scale-in-no-custom-scaledown" | ||
| soName := "so-" + deploymentName | ||
|
|
||
| // Create the scaling target. | ||
| err := k8sClient.Create(context.Background(), generateDeployment(deploymentName)) | ||
| Expect(err).ToNot(HaveOccurred()) | ||
|
|
||
| // Create the ScaledObject with custom ScaleUp behavior but NO ScaleDown behavior | ||
| // Include another annotation so that removing paused-scale-in doesn't make annotations nil | ||
| minReplicaCount := int32(1) | ||
| maxReplicaCount := int32(10) | ||
| scaleUpStabilizationWindow := int32(60) | ||
| scaleUpPeriodSeconds := int32(30) | ||
| scaleUpSelectPolicy := autoscalingv2.MaxChangePolicySelect | ||
| so := &kedav1alpha1.ScaledObject{ | ||
| ObjectMeta: metav1.ObjectMeta{ | ||
| Name: soName, | ||
| Namespace: "default", | ||
| Annotations: map[string]string{ | ||
| kedav1alpha1.PausedScaleInAnnotation: "true", | ||
| "some-other-annotation": "some-value", | ||
| }, | ||
| }, | ||
| Spec: kedav1alpha1.ScaledObjectSpec{ | ||
| ScaleTargetRef: &kedav1alpha1.ScaleTarget{ | ||
| Name: deploymentName, | ||
| }, | ||
| MinReplicaCount: &minReplicaCount, | ||
| MaxReplicaCount: &maxReplicaCount, | ||
| Triggers: []kedav1alpha1.ScaleTriggers{ | ||
| { | ||
| Type: "cron", | ||
| Metadata: map[string]string{ | ||
| "timezone": "UTC", | ||
| "start": "0 * * * *", | ||
| "end": "1 * * * *", | ||
| "desiredReplicas": "2", | ||
| }, | ||
| }, | ||
| }, | ||
| Advanced: &kedav1alpha1.AdvancedConfig{ | ||
| HorizontalPodAutoscalerConfig: &kedav1alpha1.HorizontalPodAutoscalerConfig{ | ||
| Behavior: &autoscalingv2.HorizontalPodAutoscalerBehavior{ | ||
| // Full custom ScaleUp policy, ScaleDown is nil | ||
| ScaleUp: &autoscalingv2.HPAScalingRules{ | ||
| StabilizationWindowSeconds: &scaleUpStabilizationWindow, | ||
| SelectPolicy: &scaleUpSelectPolicy, | ||
| Policies: []autoscalingv2.HPAScalingPolicy{ | ||
| { | ||
| Type: autoscalingv2.PodsScalingPolicy, | ||
| Value: 5, | ||
| PeriodSeconds: scaleUpPeriodSeconds, | ||
| }, | ||
| }, | ||
| }, | ||
| // ScaleDown is intentionally nil | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| } | ||
| err = k8sClient.Create(context.Background(), so) | ||
| Expect(err).ToNot(HaveOccurred()) | ||
| testLogger.Info("Created scaled object with custom ScaleUp but no ScaleDown behavior") | ||
|
|
||
| // Get and confirm the HPA has Disabled policy for ScaleDown due to annotation | ||
| hpa := &autoscalingv2.HorizontalPodAutoscaler{} | ||
| Eventually(func() error { | ||
| return k8sClient.Get(context.Background(), types.NamespacedName{Name: "keda-hpa-" + soName, Namespace: "default"}, hpa) | ||
| }).ShouldNot(HaveOccurred()) | ||
| Expect(hpa.Spec.Behavior).ToNot(BeNil()) | ||
| Expect(hpa.Spec.Behavior.ScaleDown).ToNot(BeNil()) | ||
| Expect(*hpa.Spec.Behavior.ScaleDown.SelectPolicy).To(Equal(autoscalingv2.DisabledPolicySelect)) | ||
| // ScaleUp should have our full custom policy | ||
| Expect(hpa.Spec.Behavior.ScaleUp).ToNot(BeNil()) | ||
| Expect(*hpa.Spec.Behavior.ScaleUp.StabilizationWindowSeconds).To(Equal(scaleUpStabilizationWindow)) | ||
| Expect(*hpa.Spec.Behavior.ScaleUp.SelectPolicy).To(Equal(scaleUpSelectPolicy)) | ||
| Expect(hpa.Spec.Behavior.ScaleUp.Policies).To(HaveLen(1)) | ||
| Expect(hpa.Spec.Behavior.ScaleUp.Policies[0].PeriodSeconds).To(Equal(scaleUpPeriodSeconds)) | ||
|
|
||
| Eventually(func() metav1.ConditionStatus { | ||
| err = k8sClient.Get(context.Background(), types.NamespacedName{Name: soName, Namespace: "default"}, so) | ||
| Ω(err).ToNot(HaveOccurred()) | ||
| return so.Status.Conditions.GetPausedCondition().Status | ||
| }, 2*time.Minute).WithPolling(5 * time.Second).Should(Equal(metav1.ConditionTrue)) | ||
|
|
||
| // Remove the annotation (DELETE, not set to false) | ||
| Eventually(func() error { | ||
| err = k8sClient.Get(context.Background(), types.NamespacedName{Name: soName, Namespace: "default"}, so) | ||
| Expect(err).ToNot(HaveOccurred()) | ||
| delete(so.ObjectMeta.Annotations, kedav1alpha1.PausedScaleInAnnotation) | ||
| return k8sClient.Update(context.Background(), so) | ||
| }).ShouldNot(HaveOccurred()) | ||
|
|
||
| Eventually(func() metav1.ConditionStatus { | ||
| err = k8sClient.Get(context.Background(), types.NamespacedName{Name: soName, Namespace: "default"}, so) | ||
| Ω(err).ToNot(HaveOccurred()) | ||
| return so.Status.Conditions.GetPausedCondition().Status | ||
| }, 2*time.Minute).WithPolling(5 * time.Second).Should(Equal(metav1.ConditionFalse)) | ||
|
|
||
| // The HPA should no longer have the Disabled ScaleDown policy | ||
| // ScaleDown.SelectPolicy should NOT be Disabled, and ScaleUp should be preserved with full custom policy | ||
| // Note: Kubernetes may default ScaleDown to some values, but it should not be Disabled | ||
| Eventually(func() bool { | ||
| err := k8sClient.Get(context.Background(), types.NamespacedName{Name: "keda-hpa-" + soName, Namespace: "default"}, hpa) | ||
| Expect(err).ToNot(HaveOccurred()) | ||
| // Behavior should still exist (for ScaleUp) | ||
| if hpa.Spec.Behavior == nil { | ||
| return false | ||
| } | ||
| // ScaleDown should NOT have Disabled policy after removing annotation | ||
| // (it may be nil or have default values, but not Disabled) | ||
| if hpa.Spec.Behavior.ScaleDown != nil && | ||
| hpa.Spec.Behavior.ScaleDown.SelectPolicy != nil && | ||
| *hpa.Spec.Behavior.ScaleDown.SelectPolicy == autoscalingv2.DisabledPolicySelect { | ||
| return false | ||
| } | ||
| // ScaleUp should still have our full custom policy | ||
| if hpa.Spec.Behavior.ScaleUp == nil { | ||
| return false | ||
| } | ||
| if hpa.Spec.Behavior.ScaleUp.StabilizationWindowSeconds == nil || | ||
| *hpa.Spec.Behavior.ScaleUp.StabilizationWindowSeconds != scaleUpStabilizationWindow { | ||
| return false | ||
| } | ||
| if hpa.Spec.Behavior.ScaleUp.SelectPolicy == nil || | ||
| *hpa.Spec.Behavior.ScaleUp.SelectPolicy != scaleUpSelectPolicy { | ||
| return false | ||
| } | ||
| if len(hpa.Spec.Behavior.ScaleUp.Policies) != 1 || | ||
| hpa.Spec.Behavior.ScaleUp.Policies[0].PeriodSeconds != scaleUpPeriodSeconds { | ||
| return false | ||
| } | ||
| return true | ||
| }).WithTimeout(1*time.Minute).WithPolling(10*time.Second).Should(BeTrue(), | ||
| "HPA ScaleDown should not be Disabled and ScaleUp should be preserved after removing paused-scale-in annotation") | ||
| }) |
There was a problem hiding this comment.
The fix in hpa.go addresses the issue for both paused-scale-in and paused-scale-out annotations, but the new test only covers the paused-scale-in case. Consider adding a similar test case for paused-scale-out annotation removal to ensure complete coverage of the fix for both annotations.
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com>
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com>
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz>
* fix: Correct parse error ActiveMQ (#7245) Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * fix: metricUnavailableValue parameter not working in Datadog scaler (#7241) * fix: metricUnavailableValue parameter not working in Datadog scaler The UseFiller flag was not being set correctly when metricUnavailableValue was configured. This fix distinguishes between 'not configured' and 'explicitly set to 0' by checking TriggerMetadata directly. Changes: - Set UseFiller in validateAPIMetadata() when metricUnavailableValue exists - Set UseFiller in validateClusterAgentMetadata() when metricUnavailableValue exists - Remove UseFiller logic from Validate() (responsibility moved to validate functions) - Update tests to verify UseFiller behavior with various values including 0 This allows users to explicitly set metricUnavailableValue to 0 and have it work as a fallback value, while still erroring when not configured. Fixes #7238 Signed-off-by: Hiroki Matsui <fenethtool@gmail.com> * test: cover both API and ClusterAgent modes in UseFiller test Updated TestDatadogMetadataValidateUseFiller to test both validateAPIMetadata() and validateClusterAgentMetadata() code paths. This ensures that the UseFiller flag is correctly set in both integration modes. Test cases now cover: - API mode: 5 test cases (not configured, 0, positive, negative, decimal) - Cluster Agent mode: 5 test cases (same variations) Signed-off-by: Hiroki Matsui <fenethtool@gmail.com> * refactor: use pointer type for FillValue to avoid TriggerMetadata access Changed FillValue from float64 to *float64 to distinguish between 'not configured' (nil) and 'explicitly set to any value including 0'. This addresses reviewer feedback about avoiding direct TriggerMetadata access and improves type safety and refactoring resistance. Changes: - FillValue type changed from float64 to *float64 with optional tag - validateAPIMetadata checks nil instead of TriggerMetadata map - validateClusterAgentMetadata checks nil instead of TriggerMetadata map - Dereference FillValue when returning fallback value (2 locations) - Update tests to handle pointer type with proper nil checks Signed-off-by: Hiroki Matsui <fenethtool@gmail.com> --------- Signed-off-by: Hiroki Matsui <fenethtool@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * Fix ScaledObject pause behavior when HPA doesn't exist (#7233) When a ScaledObject has the paused annotation set before the HPA is created, the controller would fall through and create the HPA, ignoring the pause annotation. The fix writes the paused status to etcd immediately before stopping the scale loop or deleting the HPA. This prevents race conditions where concurrent reconciles triggered by HPA deletion would not see the paused status and perform redundant operations. The key insight is to establish the paused state in etcd BEFORE any operations that trigger new reconciles, ensuring subsequent reconciles see the paused status and exit early. This solution follows the approach suggested by @rickbrouwer. Fixes #7231 Signed-off-by: nusmql <nusmql@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * fix: use TriggerError when all ScaledJob triggers fail (#7205) Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * Fix transfer-hpa-ownership panic when hpa name not provided (#7260) * chore: renormalize line endings Signed-off-by: James Williams <jamesleighwilliams@gmail.com> * fix: nil pointer when transfer-hpa-ownership is true but hpa name not specified (#7254) Signed-off-by: James Williams <jamesleighwilliams@gmail.com> * update changelog Signed-off-by: James Williams <jamesleighwilliams@gmail.com> * revert vendor changes Signed-off-by: James Williams <jamesleighwilliams@gmail.com> --------- Signed-off-by: James Williams <jamesleighwilliams@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * fix: restore HPA behavior when paused-scale-in/out annotation is deleted (#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * refactor: remove unused scaledObjectMetricSpecs variable (#7292) * refactor: remove unused scaledObjectMetricSpecs variable Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> * update CHANGELOG.md Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> --------- Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * fix: handle requestScaleLoop error in ScaledObject controller (#7273) * fix: handle requestScaleLoop error in ScaledObject controller Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> * chore: update CHANGELOG for PR #7273 Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> --------- Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> Signed-off-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * bump actions and go version (#7295) * bump actions and go version Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * bump deps Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * update pkgs Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * update tools Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * . Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * fix test Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * fix lint Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * update setup-go to use go.mod version Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * add nolint to exclude pulsar issues Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * fix devenv Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * fix codeql Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * fix splunk test Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * include job in links Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * update to ubuntu-slim some runners Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> * Update apis/keda/v1alpha1/scaledobject_webhook_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> * Update .github/workflows/scorecards.yml Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> --------- Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> Signed-off-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> * update changelog Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> --------- Signed-off-by: Rick Brouwer <rickbrouwer@gmail.com> Signed-off-by: Jorge Turrado <jorge.turrado@mail.schwarz> Signed-off-by: Hiroki Matsui <fenethtool@gmail.com> Signed-off-by: nusmql <nusmql@gmail.com> Signed-off-by: James Williams <jamesleighwilliams@gmail.com> Signed-off-by: Dima Shevchuk <dshedimon@gmail.com> Signed-off-by: u-kai <76635578+u-kai@users.noreply.github.com> Signed-off-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es> Co-authored-by: Rick Brouwer <rickbrouwer@gmail.com> Co-authored-by: Matchan <fenethtool@gmail.com> Co-authored-by: nusmql <nusmql@gmail.com> Co-authored-by: James Williams <jamesleighwilliams@gmail.com> Co-authored-by: Dima Shevchuk <dshedimon@gmail.com> Co-authored-by: Kai Udo <76635578+u-kai@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jan Wozniak <wozniak.jan@gmail.com>
|
good find! |
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com> Signed-off-by: Dmitriy Altuhov <altuhovd@gmail.com>
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com>
…ted (kedacore#7291) When paused-scale-in or paused-scale-out annotation is deleted (not set to "false") and the corresponding selectPolicy (scaleDown.selectPolicy or scaleUp.selectPolicy) is not explicitly set in the ScaledObject spec, the HPA's SelectPolicy remains stuck at "Disabled" instead of being restored. This occurs even if other behavior fields like policies or stabilizationWindowSeconds are defined - only an explicit selectPolicy value triggers the update. Root cause: DeepDerivative treats nil as "unset" and considers it a subset of any value, so DeepDerivative(nil, Disabled) returns true, preventing the HPA update. Fix: Add explicit DeepEqual check for Behavior field, following the existing pattern used for Metrics length check. test: add e2e test for paused-scale-in annotation removal Signed-off-by: Dima Shevchuk <dshedimon@gmail.com>
Summary
When
paused-scale-inorpaused-scale-outannotation is deleted (not set to"false"), the HPA'sSelectPolicyremains stuck atDisabledinstead of being restored to the default behavior. This occurs whenselectPolicyis not explicitly defined in the ScaledObject spec, even if other behavior fields likepoliciesorstabilizationWindowSecondsare configured.Problem Description
Affected Scenario
A user pauses scale-in by setting
autoscaling.keda.sh/paused-scale-in=trueon a ScaledObject. KEDA correctly sets the HPA'sscaleDown.selectPolicytoDisabled. However, when the user removes this annotation to resume normal scaling, the HPA's policy remainsDisabledindefinitely.Root Cause
The HPA update logic uses
DeepDerivativeto compare the desired HPA spec with the existing one:DeepDerivativetreatsnilas "unset" and considers it a subset of any value. So when comparing:Behavior.ScaleDown.SelectPolicy = nilBehavior.ScaleDown.SelectPolicy = DisabledDeepDerivativereturnstrue(considers them equal), so no update is triggered.How to Reproduce
paused-scale-inannotation and custom behavior whereselectPolicyis not explicitly defined, along with a dummy deployment for KEDA to scale:Disabledscale down policy:kubectl get hpa keda-hpa-my-scaledobject -o jsonpath='{.spec.behavior.scaleDown.selectPolicy}' Disabledkubectl get hpa keda-hpa-my-scaledobject -o jsonpath='{.spec.behavior.scaleDown.selectPolicy}' Disabled ← Should be defaultConditions for bug to manifest
paused-scale-in=trueorpaused-scale-out=trueannotationselectPolicydefined inspec.advanced.horizontalPodAutoscalerConfig.behavior.scaleDown(orscaleUpfor paused-scale-out). Other fields likestabilizationWindowSecondsorpoliciescan be defined - the bug still occurs."false")Solution
Add an explicit
DeepEqualcheck for theBehaviorfield before theDeepDerivativecheck:DeepEqualcorrectly identifies that{SelectPolicy: nil}and{SelectPolicy: Disabled}are different, triggering the HPA update.This follows the existing pattern already used for the
Metricslength check, which also works around aDeepDerivativelimitation.Changes
controllers/keda/hpa.goDeepEqualcheck forBehaviorfield with explanatory commentcontrollers/keda/scaledobject_controller_test.goTesting
Added integration test
removes scale down disabled policy when annotation removed without custom scaledown behaviorthat:paused-scale-in=trueannotation and customScaleUpbehavior (but noScaleDownbehavior)DisabledScaleDown policyDisabledwhile customScaleUpconfig is preserved