Skip to content

[9.2](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input#48144

Merged
kaiyan-sheng merged 3 commits into9.2from
mergify/bp/9.2/pr-47256
Dec 19, 2025
Merged

[9.2](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input#48144
kaiyan-sheng merged 3 commits into9.2from
mergify/bp/9.2/pr-47256

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Dec 17, 2025

Proposed commit message

This PR is to enhance Azure Event Hub input plugin for Elastic Agent with RBAC authorization (OAuth2) due to security requirements. Previously we only support shared access key (with connection string) for authentication.

The implementation added a new config parameter called auth_type for users to specify authentication method:
When auth_type is set to connection_string, or leave it blank: connection_string is required. When auth_typeis set toclient_secret`, oauth2 is used.

Note: We do expect users to use the same auth type for both eventhub and storage account.

OAuth2 specific Configuration Parameters (auth_type=client_secret)

When using OAuth2 authentication, the following parameters are required:

  • eventhub_namespace: Fully qualified namespace (e.g., namespace.servicebus.windows.net)
  • tenant_id: Azure AD tenant ID
  • client_id: Azure AD application (client) ID
  • client_secret: Azure AD application client secret
  • authority_host: Azure AD authority host (optional, defaults to Azure Public Cloud) https://login.microsoftonline.com is the default.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

N/A

How to test this PR locally

Setups on Azure side
  1. Setup environment variables for setting up eventhub
export RESOURCE_GROUP="kaiyan-resource-group"
export LOCATION="eastus"
export EVENTHUB_NAMESPACE="kaiyan-filebeat-test-ns"
export EVENTHUB_NAME="kaiyan-test-logs"
export STORAGE_ACCOUNT="kaiyanfbstorage"
export STORAGE_CONTAINER="kaiyan-fb-container"
export APP_NAME="filebeat-eventhub-app"
  1. Create resource group, eventhub namespace, eventhub
az group create --name $RESOURCE_GROUP --location $LOCATION
az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --location $LOCATION \
    --sku Standard
az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --namespace-name $EVENTHUB_NAMESPACE \
    --name $EVENTHUB_NAME \
    --partition-count 4
  1. Create storage account, storage container
az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --location $LOCATION \
    --sku Standard_LRS
az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT
  1. Create Azure AD application, service principle
APP_ID=$(az ad app create \
    --display-name $APP_NAME \
    --query appId --output tsv)
az ad sp create --id $APP_ID
  1. Assign eventhub role
EVENTHUB_RESOURCE_ID=$(az eventhubs namespace show \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --query id --output tsv)
az role assignment create \
    --assignee $APP_ID \
    --role "Azure Event Hubs Data Receiver" \
    --scope $EVENTHUB_RESOURCE_ID
  1. Get storage account connection string and client secret
STORAGE_CONNECTION_STRING=$(az storage account show-connection-string \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query connectionString --output tsv)
CLIENT_SECRET=$(az ad app credential reset \
    --id $APP_ID \
    --years 1 \
    --query password --output tsv)

OR
Instead of getting storage account connection string, assign storage account role:

STORAGE_RESOURCE_ID=$(az storage account show \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query id --output tsv)

az role assignment create \
    --assignee $APP_ID \
    --role "Storage Blob Data Contributor" \
    --scope $STORAGE_RESOURCE_ID
  1. Create an elastic cloud deployment and get the credentials for testing Filebeat
cloud.id: test-filebeat:foo
cloud.auth: elastic:bar
  1. Build and run Filebeat locally
mage update; mage build; ./filebeat -e
  1. Get tenant ID:
az account show --query tenantId --output tsv

When no connection_string is specified and no auth_type is specified:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns"
    tenant_id: "<redacted>"
    client_id: "<redacted>"
    client_secret: "<redacted>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_connection_string: "<redacted>"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"

We get error log when starting Filebeat:

Exiting: Failed to start crawler: starting input failed: error while initializing input: reading azure-eventhub input config: connection_string is required when auth_type is empty or set to connection_string accessing 'filebeat.inputs.0' (source:'filebeat.yml')

testing backwards compatibility:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    connection_string:  "<redacted>"
    storage_account: "kaiyanfbstorage"
    storage_account_key: "<redacted>"
    storage_account_container: "kaiyan-fb-container"

Without auth_type specified, by default we are using connection_string to keep backwards compatible. This config still works.

testing with oauth2 for both eventhub and SA:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns.servicebus.windows.net"
    tenant_id: "<your-tenant-id>" 
    client_id: "<your-app-id>"
    client_secret: "<your-secret>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"
    auth_type: "client_secret"

Screenshots

I can see logs getting ingested from Eventhub to elasticsearch with Filebeat:
Screenshot 2025-10-21 at 9 44 26 PM

Logs

I see this in the filebeat log when testing:

{"log.level":"info","@timestamp":"2025-10-28T17:28:38.082-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createContainerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":771},"message":"successfully created container client with OAuth2 authentication","service.name":"filebeat","storage_account":"kaiyanfbstorage","container":"kaiyan-fb-container","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-28T17:28:39.858-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createConsumerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":727},"message":"successfully created consumer client with OAuth2 authentication","service.name":"filebeat","namespace":"kaiyan-filebeat-test-ns.servicebus.windows.net","eventhub":"kaiyan-test-logs","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}

```<hr>This is an automatic backport of pull request #47256 done by [Mergify](https://mergify.com).

@mergify
[Azure] Add client secret (Oauth2) support for eventhub filebeat input (
8974c0e 
#47256)

(cherry picked from commit 7ffcd63)

# Conflicts:
#	x-pack/filebeat/input/azureeventhub/config.go
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Dec 17, 2025
@mergify mergify bot requested review from a team as code owners December 17, 2025 16:33
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Dec 17, 2025
@mergify mergify bot requested review from mauri870 and removed request for a team December 17, 2025 16:33
@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Dec 17, 2025

Cherry-pick of 7ffcd63 has failed:

On branch mergify/bp/9.2/pr-47256
Your branch is up to date with 'origin/9.2'.

You are currently cherry-picking commit 7ffcd634b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1761678237-feat-add-client-secret-auth-for-azure-eventhub.yaml
	modified:   docs/reference/filebeat/filebeat-input-azure-eventhub.md
	modified:   x-pack/filebeat/input/azureeventhub/README.md
	new file:   x-pack/filebeat/input/azureeventhub/auth.go
	new file:   x-pack/filebeat/input/azureeventhub/client_secret.go
	new file:   x-pack/filebeat/input/azureeventhub/client_secret_test.go
	modified:   x-pack/filebeat/input/azureeventhub/config_test.go
	modified:   x-pack/filebeat/input/azureeventhub/v2_input.go
	modified:   x-pack/filebeat/input/azureeventhub/v2_migration.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/filebeat/input/azureeventhub/config.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify bot requested a review from faec December 17, 2025 16:33
@mergify mergify bot mentioned this pull request Dec 17, 2025
Merged
6 tasks
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 17, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 17, 2025

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@github-actions github-actions bot added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Dec 17, 2025
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 17, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 17, 2025

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 17, 2025 •
edited
Loading

🔍 Preview links for changed docs

@kaiyan-sheng kaiyan-sheng merged commit aa03bec into 9.2 Dec 19, 2025
24 checks passed
@kaiyan-sheng kaiyan-sheng deleted the mergify/bp/9.2/pr-47256 branch December 19, 2025 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mauri870 mauri870 Awaiting requested review from mauri870 mauri870 is a code owner automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec faec is a code owner automatically assigned from elastic/elastic-agent-data-plane

1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport conflicts There is a conflict in the backported pull request Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elasticmachine @kaiyan-sheng