Add support for CloudWatch extended statistics (percentiles, trimmed mean, etc.) in the CloudWatch scaler to enable more advanced metric queries by adis-io · Pull Request #7109 · kedacore/keda

adis-io · 2025-09-29T09:32:22Z

Summary

Remove checkMetricStat validation and let CloudWatch validate it

The CloudWatch API will automatically recognize whether it's a standard or extended statistic based on the value

This enables using metrics like:

Percentiles: p99, p95, p50, p99.9
Trimmed Mean: TM(10:90), tm99
Trimmed Count/Sum: TC(0:100), TS(0:100)
Winsorized Mean: WM(0:100)
Percentile Rank: PR(value)
And any future extended statistics supported by CloudWatch

Testing

In main branch:

go test -v ./pkg/scalers -run TestEMFMetrics
=== RUN   TestEMFMetrics
    aws_cloudwatch_emf_test.go:94: Using AWS credentials from ~/.aws/credentials [default]
    aws_cloudwatch_emf_test.go:104: Using default AWS credential chain
    aws_cloudwatch_emf_test.go:134:
        	Error Trace:	/Users/adis-io/projects/keda/pkg/scalers/aws_cloudwatch_emf_test.go:134
        	Error:      	Received unexpected error:
        	            	error parsing cloudwatch metadata: error parsing prometheus metadata: metricStat 'p99' is not one of [SampleCount Average Sum Minimum Maximum]
        	Test:       	TestEMFMetrics
        	Messages:   	Failed to create CloudWatch scaler for EMFMetrics
--- FAIL: TestEMFMetrics (0.00s)
FAIL
FAIL	github.com/kedacore/keda/v2/pkg/scalers	1.422s
FAIL

CW_METRIC_STAT=Average go test -v ./pkg/scalers -run TestEMFMetrics
=== RUN   TestEMFMetrics
    aws_cloudwatch_emf_test.go:94: Using AWS credentials from ~/.aws/credentials [default]
    aws_cloudwatch_emf_test.go:104: Using default AWS credential chain
    aws_cloudwatch_emf_test.go:145: Metric (Average): 3.40%
    aws_cloudwatch_emf_test.go:146: Active (above 50% activation threshold): false
    aws_cloudwatch_emf_test.go:147: Target scaling value: 75%
    aws_cloudwatch_emf_test.go:148: Timestamp: 2025-09-29 12:22:34.473434 +0100 BST m=+0.611301585
    aws_cloudwatch_emf_test.go:154: ✓ Metric value 3.40% is below activation threshold (50%), scaler is inactive
--- PASS: TestEMFMetrics (0.58s)
PASS
ok  	github.com/kedacore/keda/v2/pkg/scalers	1.608s

In my branch:

go test -v ./pkg/scalers -run TestEMFMetrics
=== RUN   TestEMFMetrics
    aws_cloudwatch_emf_test.go:94: Using AWS credentials from ~/.aws/credentials [default]
    aws_cloudwatch_emf_test.go:104: Using default AWS credential chain
    aws_cloudwatch_emf_test.go:145: Metric (p99): 35.90%
    aws_cloudwatch_emf_test.go:146: Active (above 50% activation threshold): false
    aws_cloudwatch_emf_test.go:147: Target scaling value: 75%
    aws_cloudwatch_emf_test.go:148: Timestamp: 2025-09-29 12:17:48.673685 +0100 BST m=+0.678748668
    aws_cloudwatch_emf_test.go:154: ✓ Metric value 35.90% is below activation threshold (50%), scaler is inactive
--- PASS: TestEMFMetrics (0.63s)
PASS
ok  	github.com/kedacore/keda/v2/pkg/scalers	(cached)

CW_METRIC_STAT=Average go test -v ./pkg/scalers -run TestEMFMetrics
=== RUN   TestEMFMetrics
    aws_cloudwatch_emf_test.go:94: Using AWS credentials from ~/.aws/credentials [default]
    aws_cloudwatch_emf_test.go:104: Using default AWS credential chain
    aws_cloudwatch_emf_test.go:145: Metric (Average): 3.20%
    aws_cloudwatch_emf_test.go:146: Active (above 50% activation threshold): false
    aws_cloudwatch_emf_test.go:147: Target scaling value: 75%
    aws_cloudwatch_emf_test.go:148: Timestamp: 2025-09-29 12:19:18.211534 +0100 BST m=+0.622215876
    aws_cloudwatch_emf_test.go:154: ✓ Metric value 3.20% is below activation threshold (50%), scaler is inactive
--- PASS: TestEMFMetrics (0.58s)
PASS
ok  	github.com/kedacore/keda/v2/pkg/scalers	(cached)

Script for testing: https://gist.github.com/adis-io/62f6eacda900a2472a6dcc4fdf552aeb

Checklist

I have verified that my change is according to the deprecations & breaking changes policy
Changelog has been updated and is aligned with our changelog requirements
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Relates to kedacore/keda-docs#1631

github-actions · 2025-09-29T09:32:31Z

Thank you for your contribution! 🙏

Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.

While you are waiting, make sure to:

Add an entry in our changelog in alphabetical order and link related issue
Update the documentation, if needed
Add unit & e2e tests for your changes
GitHub checks are passing
Is the DCO check failing? Here is how you can fix DCO issues

Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient.

Learn more about our contribution guide.

dttung2905

sorry for a stupid question: Is there anyway we can integrate the script https://gist.github.com/adis-io/62f6eacda900a2472a6dcc4fdf552aeb into our e2e test suite? It should be put under https://github.com/kedacore/keda/tree/main/tests

adis-io · 2025-09-30T21:11:26Z

sorry for a stupid question: Is there anyway we can integrate the script https://gist.github.com/adis-io/62f6eacda900a2472a6dcc4fdf552aeb into our e2e test suite? It should be put under https://github.com/kedacore/keda/tree/main/tests

Hey, it's a good idea. ~~If I add or adapt existing tests, can I run it in my cluster or it only works with /run-e2e aws_cloudwatch?~~ Tested in my cluster already

tests/scalers/aws/aws_cloudwatch_metric_stat/aws_cloudwatch_metric_stat_test.go

adis-io · 2025-10-01T14:20:18Z

hey @dttung2905, I've created a new test based on tests/scalers/aws/aws_cloudwatch/aws_cloudwatch_test.go

Please let me know if I need to do anything else

zroubalik · 2025-10-02T13:50:19Z

/run-e2e aws
Update: You can check the progress here

zroubalik

@adis-io could you please update documentation for this scaler to mention this?

Also, does this breaks any exsiting setup?

rickbrouwer · 2025-10-02T13:52:27Z

There is also a merge conflict for this PR

adis-io · 2025-10-02T15:44:58Z

@adis-io could you please update documentation for this scaler to mention this?

kedacore/keda-docs#1631

Also, does this breaks any exsiting setup?

I guess there were no need to add constraint as CW supports all these metric stats in the first place hence no breaking change

There is also a merge conflict for this PR

Rebased

rickbrouwer · 2025-10-02T15:52:38Z

/run-e2e aws
Update: You can check the progress here

adis-io · 2025-10-02T16:50:02Z

/run-e2e aws Update: You can check the progress here

Could you retry e2e tests please?

I've accidentally pushed different version of test file, now should be resolved (locally I was testing with short lived credentials to push metrics and pod identity to fetch them)

rickbrouwer · 2025-10-02T16:53:17Z

/run-e2e aws
Update: You can check the progress here

adis-io · 2025-10-03T08:20:51Z

@rickbrouwer hey sorry for bothering, any chance you can restart the build? Seems like it's either stuck or flaky test

rickbrouwer · 2025-10-03T09:29:29Z

/run-e2e aws
Update: You can check the progress here

dttung2905

LGTM! Thanks for the contribution 🙇

CHANGELOG.md

JorTurFer · 2025-10-04T10:56:19Z

Looking nice! 🙇 only one comment here inline and another in docs.

The CloudWatch API will automatically recognize whether it's a standard or extended statistic based on the value This enables using metrics like: - Percentiles: p99, p95, p50, p99.9 - Trimmed Mean: TM(10:90), tm99 - Trimmed Count/Sum: TC(0:100), TS(0:100) - Winsorized Mean: WM(0:100) - Percentile Rank: PR(value) - And any future extended statistics supported by CloudWatch Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

JorTurFer · 2025-10-04T12:26:30Z

thanks a lot for the contribution!

JorTurFer · 2025-10-04T12:26:37Z

/run-e2e aws
Update: You can check the progress here

…mean, etc.) in the CloudWatch scaler to enable more advanced metric queries (kedacore#7109) * Remove checkMetricStat validation and let CloudWatch validate it The CloudWatch API will automatically recognize whether it's a standard or extended statistic based on the value This enables using metrics like: - Percentiles: p99, p95, p50, p99.9 - Trimmed Mean: TM(10:90), tm99 - Trimmed Count/Sum: TC(0:100), TS(0:100) - Winsorized Mean: WM(0:100) - Percentile Rank: PR(value) - And any future extended statistics supported by CloudWatch Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Update CHANGELOG.md Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Add E2E test for using non default MetricStat Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Fix e2e test Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Update CHANGELOG.md Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> --------- Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Dmitriy Altuhov <altuhovd@gmail.com>

…mean, etc.) in the CloudWatch scaler to enable more advanced metric queries (kedacore#7109) * Remove checkMetricStat validation and let CloudWatch validate it The CloudWatch API will automatically recognize whether it's a standard or extended statistic based on the value This enables using metrics like: - Percentiles: p99, p95, p50, p99.9 - Trimmed Mean: TM(10:90), tm99 - Trimmed Count/Sum: TC(0:100), TS(0:100) - Winsorized Mean: WM(0:100) - Percentile Rank: PR(value) - And any future extended statistics supported by CloudWatch Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Update CHANGELOG.md Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Add E2E test for using non default MetricStat Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Fix e2e test Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> * Update CHANGELOG.md Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> --------- Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com> Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es>

keda-automation requested review from a team September 29, 2025 09:32

adis-io force-pushed the patch-1 branch 5 times, most recently from 9377002 to a3a70e6 Compare September 29, 2025 12:27

adis-io marked this pull request as ready for review September 29, 2025 12:27

dttung2905 reviewed Sep 29, 2025

View reviewed changes

adis-io force-pushed the patch-1 branch from a3a70e6 to f5865bc Compare October 1, 2025 14:16

keda-automation requested a review from a team October 1, 2025 14:16

semgrep-app bot reviewed Oct 1, 2025

View reviewed changes

tests/scalers/aws/aws_cloudwatch_metric_stat/aws_cloudwatch_metric_stat_test.go Show resolved Hide resolved

adis-io force-pushed the patch-1 branch from f5865bc to 62ea38f Compare October 1, 2025 14:18

adis-io force-pushed the patch-1 branch from 62ea38f to f53ecff Compare October 1, 2025 15:48

zroubalik reviewed Oct 2, 2025

View reviewed changes

adis-io force-pushed the patch-1 branch from f53ecff to 6c62d03 Compare October 2, 2025 15:23

keda-automation requested a review from a team October 2, 2025 15:24

adis-io mentioned this pull request Oct 2, 2025

Add documentation for MetricStat usage with aws-cloudwatch kedacore/keda-docs#1631

Merged

1 task

adis-io force-pushed the patch-1 branch from f9ba992 to ff4d6ec Compare October 2, 2025 16:47

zroubalik approved these changes Oct 3, 2025

View reviewed changes

zroubalik mentioned this pull request Oct 3, 2025

Release: 2.18.0 #6853

Closed

22 tasks

dttung2905 approved these changes Oct 3, 2025

View reviewed changes

rickbrouwer added the ok-to-merge This PR can be merged label Oct 3, 2025

JorTurFer reviewed Oct 4, 2025

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

keda-automation requested review from a team October 4, 2025 11:45

adis-io and others added 5 commits October 4, 2025 12:46

Update CHANGELOG.md

38dce88

Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

Add E2E test for using non default MetricStat

66fc36f

Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

Fix e2e test

8d11bc7

Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

Update CHANGELOG.md

1175345

Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es> Signed-off-by: Adis Osmonov <adis.osmonov@gmail.com>

adis-io force-pushed the patch-1 branch from 531ed5d to 1175345 Compare October 4, 2025 11:46

JorTurFer approved these changes Oct 4, 2025

View reviewed changes

JorTurFer merged commit 392c23a into kedacore:main Oct 4, 2025
24 checks passed

Conversation

adis-io commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Checklist

Uh oh!

github-actions bot commented Sep 29, 2025

Uh oh!

dttung2905 left a comment

Choose a reason for hiding this comment

Uh oh!

adis-io commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

adis-io commented Oct 1, 2025

Uh oh!

zroubalik commented Oct 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

rickbrouwer commented Oct 2, 2025

Uh oh!

adis-io commented Oct 2, 2025

Uh oh!

rickbrouwer commented Oct 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adis-io commented Oct 2, 2025

Uh oh!

rickbrouwer commented Oct 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adis-io commented Oct 3, 2025

Uh oh!

rickbrouwer commented Oct 3, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dttung2905 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JorTurFer commented Oct 4, 2025

Uh oh!

JorTurFer commented Oct 4, 2025

Uh oh!

JorTurFer commented Oct 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adis-io commented Sep 29, 2025 •

edited

Loading

adis-io commented Sep 30, 2025 •

edited

Loading

zroubalik commented Oct 2, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 2, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 2, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 3, 2025 •

edited by github-actions bot

Loading

JorTurFer commented Oct 4, 2025 •

edited by github-actions bot

Loading