Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Current EventHub output works great - for some users there is a need to control how events get sent to specific partitions for event ordering or other similar processing requirements

The Azure Eventhubs Go module which the output plugin uses suggests that this could/should be as simple as setting the PartitionKey value for an event and then eventhub will take care of hashing based on that key and consistently send events with that particular value to the same partition. - https://github.com/Azure/azure-event-hubs-go#send-and-receive

I propose that we add some configuration that allows the user to configure which value to set for this based on a specific Tag or data field with the default to omit this entirely which would retain current behaviour by default.

e.g.

partition_key_field = "SubscriberId"

Current behavior:

No partition key value is set on events meaning all events are randomly load balanced over eventhub partitions

Desired behavior:

PartitionKey value gets set on events so that they are consistently load balanced over eventhub partitions in a deterministic manner

Use case:

This is useful because by default downstream processing can then take advantage that specific events will group into specific partitions and can perform relevant optimizations accordingly. an example in this scenario is Azure Stream Analytics Jobs

Also important to note is that some events may be very order dependent - being able to control events going to specific partitions allows for order of events to be guaranteed in subsequent processing