Use approximate queue length for azure queue scaler#4003
Use approximate queue length for azure queue scaler#4003JorTurFer merged 6 commits intokedacore:mainfrom
Conversation
Signed-off-by: Yue Fei <yufei@microsoft.com>
8d0366e to
d18575f
Compare
|
By removing the visible count you are not actually achieving what you are suggesting in #4002 |
Removing the visible count will return the approximate message count directly, which contains messages that are queued and being processed. Not sure if there is anything wrong? @tomkerkhove |
|
/run-e2e azure_queue |
|
I have misread the issue, no worries. |
|
Can you add an entry in the changelog please? I've pinged @ahmelsayed privately as well to see if we can remember why we added that. |
Signed-off-by: Yue Fei <59813791+moria97@users.noreply.github.com>
|
@ahmelsayed pointed out that this was changed in #2613 by @RamCohen as an optimization. I think the optimization is still OK to have and we can skip this PR but I don't have a strong opinion. |
|
@tomkerkhove This optimization looks make sense to me. But the issue is not related to the optimization, but the visible message count used to count queue length. Let's say "queue length" parameter is 1, there are 20 messages and scaled out to 20 instances and pulled message from queue, the queue size returned is 0 (visible message account), but the actual value should be 20 (approximate queue length). |
zroubalik
left a comment
There was a problem hiding this comment.
@moria97 @tomkerkhove @v-shenoy so what is the status of this PR?
|
I'm shipping a custom version of this scaler in production since v2.6.1. My custom versions have the exact same code as in this pull request, and they have been working perfectly fine for the past year. ApproximateMessageCount is the way to go. The optimization done with #2613 will be useless, because we should always call GetProperties to get the ApproximateMessageCount. I hope this will be merged soon. |
Signed-off-by: Yue Fei <59813791+moria97@users.noreply.github.com>
|
@tomkerkhove @zroubalik Can you take a look at this pr? |
|
/run-e2e azure_queue Update: You can check the progress here |
Signed-off-by: Yue Fei <59813791+moria97@users.noreply.github.com>
Signed-off-by: Yue Fei <59813791+moria97@users.noreply.github.com>
|
/run-e2e azure_queue* |
|
Thanks team, this is great! |
Use the approximate queue length instead of visible messages, see #4002
Checklist
Fixes #4002
Relates to #