[8.19](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input by mergify[bot] · Pull Request #48142 · elastic/beats

mergify · 2025-12-17T16:33:31Z

Proposed commit message

This PR is to enhance Azure Event Hub input plugin for Elastic Agent with RBAC authorization (OAuth2) due to security requirements. Previously we only support shared access key (with connection string) for authentication.

The implementation added a new config parameter called auth_type for users to specify authentication method:
When auth_type is set to connection_string, or leave it blank: connection_string is required. When auth_typeis set toclient_secret`, oauth2 is used.

Note: We do expect users to use the same auth type for both eventhub and storage account.

OAuth2 specific Configuration Parameters (`auth_type`=`client_secret`)

When using OAuth2 authentication, the following parameters are required:

eventhub_namespace: Fully qualified namespace (e.g., namespace.servicebus.windows.net)
tenant_id: Azure AD tenant ID
client_id: Azure AD application (client) ID
client_secret: Azure AD application client secret
authority_host: Azure AD authority host (optional, defaults to Azure Public Cloud) https://login.microsoftonline.com is the default.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

N/A

How to test this PR locally

Setups on Azure side

Setup environment variables for setting up eventhub

export RESOURCE_GROUP="kaiyan-resource-group"
export LOCATION="eastus"
export EVENTHUB_NAMESPACE="kaiyan-filebeat-test-ns"
export EVENTHUB_NAME="kaiyan-test-logs"
export STORAGE_ACCOUNT="kaiyanfbstorage"
export STORAGE_CONTAINER="kaiyan-fb-container"
export APP_NAME="filebeat-eventhub-app"

Create resource group, eventhub namespace, eventhub

az group create --name $RESOURCE_GROUP --location $LOCATION
az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --location $LOCATION \
    --sku Standard
az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --namespace-name $EVENTHUB_NAMESPACE \
    --name $EVENTHUB_NAME \
    --partition-count 4

Create storage account, storage container

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --location $LOCATION \
    --sku Standard_LRS
az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT

Create Azure AD application, service principle

APP_ID=$(az ad app create \
    --display-name $APP_NAME \
    --query appId --output tsv)
az ad sp create --id $APP_ID

Assign eventhub role

EVENTHUB_RESOURCE_ID=$(az eventhubs namespace show \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --query id --output tsv)
az role assignment create \
    --assignee $APP_ID \
    --role "Azure Event Hubs Data Receiver" \
    --scope $EVENTHUB_RESOURCE_ID

Get storage account connection string and client secret

STORAGE_CONNECTION_STRING=$(az storage account show-connection-string \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query connectionString --output tsv)
CLIENT_SECRET=$(az ad app credential reset \
    --id $APP_ID \
    --years 1 \
    --query password --output tsv)

OR
Instead of getting storage account connection string, assign storage account role:

STORAGE_RESOURCE_ID=$(az storage account show \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query id --output tsv)

az role assignment create \
    --assignee $APP_ID \
    --role "Storage Blob Data Contributor" \
    --scope $STORAGE_RESOURCE_ID

Create an elastic cloud deployment and get the credentials for testing Filebeat

cloud.id: test-filebeat:foo
cloud.auth: elastic:bar

Build and run Filebeat locally

mage update; mage build; ./filebeat -e

Get tenant ID:

az account show --query tenantId --output tsv

When no `connection_string` is specified and no `auth_type` is specified:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns"
    tenant_id: "<redacted>"
    client_id: "<redacted>"
    client_secret: "<redacted>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_connection_string: "<redacted>"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"

We get error log when starting Filebeat:

Exiting: Failed to start crawler: starting input failed: error while initializing input: reading azure-eventhub input config: connection_string is required when auth_type is empty or set to connection_string accessing 'filebeat.inputs.0' (source:'filebeat.yml')

testing backwards compatibility:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    connection_string:  "<redacted>"
    storage_account: "kaiyanfbstorage"
    storage_account_key: "<redacted>"
    storage_account_container: "kaiyan-fb-container"

Without auth_type specified, by default we are using connection_string to keep backwards compatible. This config still works.

testing with oauth2 for both eventhub and SA:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns.servicebus.windows.net"
    tenant_id: "<your-tenant-id>" 
    client_id: "<your-app-id>"
    client_secret: "<your-secret>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"
    auth_type: "client_secret"

Screenshots

I can see logs getting ingested from Eventhub to elasticsearch with Filebeat:

Logs

I see this in the filebeat log when testing:

{"log.level":"info","@timestamp":"2025-10-28T17:28:38.082-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createContainerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":771},"message":"successfully created container client with OAuth2 authentication","service.name":"filebeat","storage_account":"kaiyanfbstorage","container":"kaiyan-fb-container","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-28T17:28:39.858-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createConsumerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":727},"message":"successfully created consumer client with OAuth2 authentication","service.name":"filebeat","namespace":"kaiyan-filebeat-test-ns.servicebus.windows.net","eventhub":"kaiyan-test-logs","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}

```<hr>This is an automatic backport of pull request #47256 done by [Mergify](https://mergify.com).

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # docs/reference/filebeat/filebeat-input-azure-eventhub.md # x-pack/filebeat/input/azureeventhub/config.go

mergify · 2025-12-17T16:33:34Z

Cherry-pick of 7ffcd63 has failed:

On branch mergify/bp/8.19/pr-47256
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 7ffcd634b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1761678237-feat-add-client-secret-auth-for-azure-eventhub.yaml
	modified:   x-pack/filebeat/input/azureeventhub/README.md
	new file:   x-pack/filebeat/input/azureeventhub/auth.go
	new file:   x-pack/filebeat/input/azureeventhub/client_secret.go
	new file:   x-pack/filebeat/input/azureeventhub/client_secret_test.go
	modified:   x-pack/filebeat/input/azureeventhub/config_test.go
	modified:   x-pack/filebeat/input/azureeventhub/v2_input.go
	modified:   x-pack/filebeat/input/azureeventhub/v2_migration.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   docs/reference/filebeat/filebeat-input-azure-eventhub.md
	both modified:   x-pack/filebeat/input/azureeventhub/config.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

elasticmachine · 2025-12-17T16:33:47Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

github-actions · 2025-12-17T16:33:48Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…47256

belimawr

@kaiyan-sheng is it intentional not to document it in 8.19?

kaiyan-sheng · 2025-12-18T18:30:45Z

Hey @belimawr 👋 It's not 😄 The doc is in a diff file for 8.19 branch I think. I will make the change and push it into the next commit!

belimawr · 2025-12-18T19:56:36Z

Hey @belimawr 👋 It's not 😄 The doc is in a diff file for 8.19 branch I think. I will make the change and push it into the next commit!

Yeah, 8.19 is still using the old asciidoc system, so when backporting PRs docs need to be manually migrated to the old system.

[Azure] Add client secret (Oauth2) support for eventhub filebeat input (

dad73be

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # docs/reference/filebeat/filebeat-input-azure-eventhub.md # x-pack/filebeat/input/azureeventhub/config.go

mergify bot added backport conflicts There is a conflict in the backported pull request labels Dec 17, 2025

mergify bot requested review from a team as code owners December 17, 2025 16:33

mergify bot assigned kaiyan-sheng Dec 17, 2025

mergify bot requested review from belimawr and leehinman and removed request for a team December 17, 2025 16:33

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 17, 2025

mergify bot mentioned this pull request Dec 17, 2025

[Azure] Add client secret (Oauth2) support for eventhub filebeat input #47256

Merged

6 tasks

github-actions bot added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Dec 17, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 17, 2025

kaiyan-sheng added 3 commits December 17, 2025 13:03

Merge remote-tracking branch 'upstream/8.19' into mergify/bp/8.19/pr-…

635df91

…47256

remove doc

2a35139

remove merge conflict

89bdc90

belimawr approved these changes Dec 18, 2025

View reviewed changes

add doc

8916c00

kaiyan-sheng merged commit 668fd61 into 8.19 Dec 18, 2025
39 checks passed

kaiyan-sheng deleted the mergify/bp/8.19/pr-47256 branch December 18, 2025 23:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.19](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input#48142

[8.19](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input#48142
kaiyan-sheng merged 5 commits into8.19from
mergify/bp/8.19/pr-47256

mergify bot commented Dec 17, 2025

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

elasticmachine commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

belimawr left a comment

Uh oh!

kaiyan-sheng commented Dec 18, 2025

Uh oh!

belimawr commented Dec 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mergify bot commented Dec 17, 2025

Proposed commit message

OAuth2 specific Configuration Parameters (auth_type=client_secret)

Checklist

Disruptive User Impact

How to test this PR locally

When no connection_string is specified and no auth_type is specified:

testing backwards compatibility:

testing with oauth2 for both eventhub and SA:

Screenshots

Logs

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

elasticmachine commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

🤖 GitHub comments

Uh oh!

belimawr left a comment

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng commented Dec 18, 2025

Uh oh!

belimawr commented Dec 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

OAuth2 specific Configuration Parameters (`auth_type`=`client_secret`)

When no `connection_string` is specified and no `auth_type` is specified: