Proposed commit message

Restructure the azure-eventhub input, rebranding the current version as processor v1. Add a brand new processor v2, allowing users to select which version to use in the config:

• processor v1: uses the legacy Event Hub SDK (default processor, at least for 8.15)
• processor v2: uses the modern Event Hub SDK

Why are we introducing a processor v2?

• processor v1 uses deprecated libraries
  • github.com/Azure/azure-event-hubs-go (legacy)
  • github.com/Azure/azure-storage-blob-go (legacy, retiring on Sep 2024)
• processor v1 does not support publishing acks (mostly due to lack of hooks; the legacy SDK is a black box)

Notes for reviewers

Overview

To help with the review, here is an overview of the main flow of the processor v2-based input.

• The processor v2 starts a new consumer for each event hub partition.
• Each consumer creates a pipeline client.
• When a consumer receives an event, it decodes it and sends it to the pipeline client.
• When the pipeline successfully processes the event, it acknowledges with the consumer.
• The consumer stores the sequence number of the last successful event in the partition blob in the storage account container.

New features

• Replace the legacy SDK with the new modern and supported SDK
• Add support for publishing ACKs
• Add a migration assistant to migrate checkpoint v1 information to the v2 format

Replace the legacy SDK with the new modern and supported SDK

The new SDK is more flexible and allows us to implement new features and configuration options.

Add support for publishing ACKs

Now, the processor v2 updates the sequence number only when the events have been successfully delivered to Elasticsearch.

Add a migration assistant to migrate checkpoint v1 information to the v2 format

On the first start of the processor v2, the migration assistant (enabled by default) checks if checkpoint v1 information exists from processor v1 and migrates them to the v2 format.

See "Scenario 001: Migration" at x-pack/filebeat/input/azureeventhub/README.md for more details.

New configuration options

There are new configuration options for v2:

• storage_account_connection_string (required) to authenticate with the storage account container.
• migrate_checkpoint (optional, default: yes) controls if the processor v2 should check and migrate checkpoint v1 information on start.
• processor_version (optional, default: v1) which processor version to use.
• processor_update_interval (optional, default: 10s) time interval between checking if new partitions are available.
• processor_start_position (optional, default: earliest) controls if the processor should start from the beginning earliest or the latest event in the event hub retention period.
• partition_receive_timeout (optional, default: 5s)
• partition_receive_count (optional, default: 100)

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

How to test this PR locally

See "Test Scenarios" section in the x-pack/filebeat/input/azureeventhub/README.md file.

Related issues

• Closes [Azure] Migrate the azure-eventhub input to the new Azure Event Hubs Client Module for Go #33815

Use cases

Screenshots

Logs

### Author's Checklist
- [ ] Test that the `migrationAssistant` returns a "failed to download checkpoint v1 information for partition"
- [ ] Test that the `migrationAssistant` returns a "failed to parse offset"
- [ ] Compare Storage Account metrics (transactions, blob capacity, blob count, egress, ingress) between v1 and v2
- [ ] Elastic Agent build
- [ ] Performance testing with 1K, 10K, 100K, 1M log events
```<hr>This is an automatic backport of pull request #39796 done by [Mergify](https://mergify.com).