Azure Logs: Expand the integration docs adding a section for Event Hub partitions#10583
Conversation
0f9c076 to
748c725
Compare
🚀 Benchmarks reportTo see the full report comment with |
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
|
|
||
| #### How many partitions? | ||
|
|
||
| The number of partitions is essential to balance Event Hub cost and performance. |
There was a problem hiding this comment.
As we represent the Event Hub in different cases in the multiple places. Some places we do have the Event hub. To make this consistent, I would recommend to keep lower case in the descriptions and the titles as Event Hub.
There was a problem hiding this comment.
Good point. Let me review the doc to improve consistency!
There was a problem hiding this comment.
After reading the following docs:
- https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about
- https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features
@alaudazzi @muthu-mps, I'm reviewing the _dev/docs/README.md following these rules:
- "Event Hubs" for the service name
- "event hub" for the actual event hub
There was a problem hiding this comment.
After reading https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings, I'm also updating the capitalization of diagnostic settings.
There was a problem hiding this comment.
I'm not 100% sure about how to capitalize the names in the diagrams. @alaudazzi, let me know what you think!
alaudazzi
left a comment
There was a problem hiding this comment.
Left a few minor editing suggestions, other LGTM.
More generally, you might want to check if event hub is capitalized consistently.
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
💚 Build Succeeded
History
cc @zmoog |
|
|
|
||
| With more than one Agent, setting the number of partitions is crucial. The agents share the existing partitions to scale out performance and improve availability. | ||
|
|
||
| The number of partitions must be at least the number of agents. |
There was a problem hiding this comment.
looking to the diagram below - is it possible that first agent when starting create workers for all partitions, so when second agent starts - there are no "free" partitions?
correctly I understand that it is not possible that workers of different agents are picking the same partition?
There was a problem hiding this comment.
The agents use blobs in a storage account to share two kinds of information: partition ownership 1 and checkpoint information.
For example, when the input starts:
- it leases the blob for the partition
- reads the checkpoint info
- and sets the ownership.
If there's another input from another agent using the same agent policy, this input knows that another input claimed that partition and moves to the next partition.
The processor v2 (thanks to the modern event hub SDK) offers a configurable load balancing strategy for partition claiming.
From sdk/messaging/azeventhubs/processor.go;
ProcessorStrategyBalanced will attempt to claim a single partition at a time, until each active owner has an equal share of partitions.
Footnotes
-
The implementation details differ between processors v1 and v2. Processor v1 stores the ownership information in a JSON document, while processor v2 stores it as blob metadata. ↩
There was a problem hiding this comment.
Thanks for the detailed explanation!
|
Package azure - 1.13.1 containing this change is available at https://epr.elastic.co/search?package=azure |
…b partitions (elastic#10583) Extend the Azure Logs integration docs, adding a section dedicated to setting up event hub partitions. The existing documentation didn't cover event hub partitions, leaving the responsibility of reading the Microsoft documentation. This PR gives guidance on the event hub partitions, linking to Microsoft docs for advanced use cases. --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
…b partitions (elastic#10583) Extend the Azure Logs integration docs, adding a section dedicated to setting up event hub partitions. The existing documentation didn't cover event hub partitions, leaving the responsibility of reading the Microsoft documentation. This PR gives guidance on the event hub partitions, linking to Microsoft docs for advanced use cases. --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co> Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
Proposed commit message
Extend the Azure Logs integration docs adding a section dedicated to setting up event hub partitions.
The existing docs didn't cover event hub partitions, leaving the responsibility of reading the Microsoft documentation. This PR gives guidance on the event hub partitions, linking to Microsoft docs for advanced use cases.
Checklist
I have verified that all data streams collect metrics or logs.changelog.ymlfile.I have verified that Kibana version constraints are current according to guidelines.Related issues