Proposal

Currently RabbitMQ scaler allows to use QueueLength and MessageRate trigger modes. I would like to expand this list with three additional triggers:

• DeliverGetRate, which will activate, when given messages delivery rate will be reached
• PublishedToDeliveredRatio, which will activate, when scaler hits certain level of published to delivered messages ratio
• ExpectedQueueConsumptionTime, which will be activated when the total estimated waiting time for delivery of all messages available in the queue at a given moment (with the current capacity of consumers) reaches the indicated value.

Use-Case

I work for a small software house company. We have this part of the system, where dedicated pool of consumer instances fetches messages from queue for further processing, and we want consumers to scale within reasonable boundaries and only when it would be really needed. Using only the number of available messages in the queue and messages publishing ratio is for us insufficient, since having variable stats for publishing and consuming messages during the day (along with couple other external variable factors) there is no way to tell the scaler to stop scale-out process, or when to gracefully perform scale-in task. That said, having mentioned triggers at hand we can have additional, more sophisticated control over scale-out and scale-in operations, e.g:

• with PublishedToDeliveredRatio trigger - if publishing rate is equal to consumption rate at the very moment, then it is the indicator that capacity of consumers is sufficient for handling ongoing messages processing and no additional resources need to be allocated; if ratio is below 1, KEDA can gracefully take down some consumers but keep stable messages processing rate
• with ExpectedQueueConsumptionTime trigger - if delivery time of all messages in the queue using all available consmer pods at the very moment exceeds expected deadline, we can scale-out consuming service to keep the performance level up to specs.

With all those new triggers and some additional math in scalingModifiers we can also ensure smooth scaling (which could take into account dynamically calculated hysteresis and consumer's capacity) and protect additional resources (like databases with processed content from the messages) from overloading, i.e. once the delivery rate will hit certain level and database engine will suffer from further grow of consumers processing capacity, or the performance of the consumers will drop due of some incident, we can just stop scaling-out processing replicas and trigger scale-in operation if needed.

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

Those modes are being used by us for at least 3 month (with custom KEDA images, using DeliverGetRate and PublishedToDeliveredRatio; last couple of weeks including lately added ExpectedQueueConsumptionTime) and they perform exceptionally well, considering our needs.

We're fully aware, that implementing only DeliverGetRate should suffice our needs, but having also built-in PublishedToDeliveredRatio and ExpectedQueueConsumptionTime trigger modes without manually defining additional math behind them within scalingModifiers blocks was a low hanging fruit not to be ignored.