[Azure] Add client secret (Oauth2) support for eventhub filebeat input by kaiyan-sheng · Pull Request #47256 · elastic/beats

kaiyan-sheng · 2025-10-22T04:26:17Z

Proposed commit message

This PR is to enhance Azure Event Hub input plugin for Elastic Agent with RBAC authorization (OAuth2) due to security requirements. Previously we only support shared access key (with connection string) for authentication.

The implementation added a new config parameter called auth_type for users to specify authentication method:
When auth_type is set to connection_string, or leave it blank: connection_string is required. When auth_typeis set toclient_secret`, oauth2 is used.

Note: We do expect users to use the same auth type for both eventhub and storage account.

OAuth2 specific Configuration Parameters (`auth_type`=`client_secret`)

When using OAuth2 authentication, the following parameters are required:

eventhub_namespace: Fully qualified namespace (e.g., namespace.servicebus.windows.net)
tenant_id: Azure AD tenant ID
client_id: Azure AD application (client) ID
client_secret: Azure AD application client secret
authority_host: Azure AD authority host (optional, defaults to Azure Public Cloud) https://login.microsoftonline.com is the default.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

N/A

How to test this PR locally

Setups on Azure side

Setup environment variables for setting up eventhub

export RESOURCE_GROUP="kaiyan-resource-group"
export LOCATION="eastus"
export EVENTHUB_NAMESPACE="kaiyan-filebeat-test-ns"
export EVENTHUB_NAME="kaiyan-test-logs"
export STORAGE_ACCOUNT="kaiyanfbstorage"
export STORAGE_CONTAINER="kaiyan-fb-container"
export APP_NAME="filebeat-eventhub-app"

Create resource group, eventhub namespace, eventhub

az group create --name $RESOURCE_GROUP --location $LOCATION
az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --location $LOCATION \
    --sku Standard
az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --namespace-name $EVENTHUB_NAMESPACE \
    --name $EVENTHUB_NAME \
    --partition-count 4

Create storage account, storage container

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --location $LOCATION \
    --sku Standard_LRS
az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT

Create Azure AD application, service principle

APP_ID=$(az ad app create \
    --display-name $APP_NAME \
    --query appId --output tsv)
az ad sp create --id $APP_ID

Assign eventhub role

EVENTHUB_RESOURCE_ID=$(az eventhubs namespace show \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --query id --output tsv)
az role assignment create \
    --assignee $APP_ID \
    --role "Azure Event Hubs Data Receiver" \
    --scope $EVENTHUB_RESOURCE_ID

Get storage account connection string and client secret

STORAGE_CONNECTION_STRING=$(az storage account show-connection-string \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query connectionString --output tsv)
CLIENT_SECRET=$(az ad app credential reset \
    --id $APP_ID \
    --years 1 \
    --query password --output tsv)

OR
Instead of getting storage account connection string, assign storage account role:

STORAGE_RESOURCE_ID=$(az storage account show \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query id --output tsv)

az role assignment create \
    --assignee $APP_ID \
    --role "Storage Blob Data Contributor" \
    --scope $STORAGE_RESOURCE_ID

Create an elastic cloud deployment and get the credentials for testing Filebeat

cloud.id: test-filebeat:foo
cloud.auth: elastic:bar

Build and run Filebeat locally

mage update; mage build; ./filebeat -e

Get tenant ID:

az account show --query tenantId --output tsv

When no `connection_string` is specified and no `auth_type` is specified:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns"
    tenant_id: "<redacted>"
    client_id: "<redacted>"
    client_secret: "<redacted>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_connection_string: "<redacted>"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"

We get error log when starting Filebeat:

Exiting: Failed to start crawler: starting input failed: error while initializing input: reading azure-eventhub input config: connection_string is required when auth_type is empty or set to connection_string accessing 'filebeat.inputs.0' (source:'filebeat.yml')

testing backwards compatibility:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    connection_string:  "<redacted>"
    storage_account: "kaiyanfbstorage"
    storage_account_key: "<redacted>"
    storage_account_container: "kaiyan-fb-container"

Without auth_type specified, by default we are using connection_string to keep backwards compatible. This config still works.

testing with oauth2 for both eventhub and SA:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns.servicebus.windows.net"
    tenant_id: "<your-tenant-id>" 
    client_id: "<your-app-id>"
    client_secret: "<your-secret>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"
    auth_type: "client_secret"

Screenshots

I can see logs getting ingested from Eventhub to elasticsearch with Filebeat:

Logs

I see this in the filebeat log when testing:

{"log.level":"info","@timestamp":"2025-10-28T17:28:38.082-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createContainerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":771},"message":"successfully created container client with OAuth2 authentication","service.name":"filebeat","storage_account":"kaiyanfbstorage","container":"kaiyan-fb-container","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-28T17:28:39.858-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createConsumerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":727},"message":"successfully created consumer client with OAuth2 authentication","service.name":"filebeat","namespace":"kaiyan-filebeat-test-ns.servicebus.windows.net","eventhub":"kaiyan-test-logs","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}

github-actions · 2025-10-22T04:26:32Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-10-22T04:26:54Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

github-actions · 2025-10-22T04:28:02Z

🔍 Preview links for changed docs

docs/reference/filebeat/filebeat-input-azure-eventhub.md

colleenmcginnis

Starting with v9.0, there is no longer a new documentation set published with every minor release: the same page stays valid over time and shows version-related evolutions. Read more in Write cumulative documentation.

Based on the backport labels, it looks like these changes are likely targeting 9.3.0. If that's the case, you can use my suggestions below. If not, feel free to adjust my suggestions as needed.

changelog/fragments/1761678237-feat-add-oauth2-for-azure-eventhub.yaml

docs/reference/filebeat/filebeat-input-azure-eventhub.md

elasticmachine · 2025-10-30T14:22:35Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

kaiyan-sheng · 2025-10-30T16:15:35Z

@colleenmcginnis Thank you so much for the comments! Yes this change will likely to go in for 9.3.0. Definitely no backports since it is a feature.

zmoog

I've reviewed the changes in the oauth2_azure branch. This is a solid first PR for adding OAuth2 authentication to the Azure Event Hub input.

Here are my observations.

I would move the authentication-related code into dedicated files, with helper function(s) that takes the config and returns a azcore.TokenCredential — or a client, since the current legacy authentication methods are not included in azidentity.

// Just a high-level example
func createAzureCredential(config azureInputConfig, log *logp.Logger) (azcore.TokenCredential, error) {
      credentialOptions := &azidentity.ClientSecretCredentialOptions{
          ClientOptions: azcore.ClientOptions{
              Cloud: getAzureCloud(config.AuthorityHost),
          },
      }

      return azidentity.NewClientSecretCredential(
          config.TenantID,
          config.ClientID,
          config.ClientSecret,
          credentialOptions,
      )
  }

I'm usually against premature optimizations, but in this case we have a list of authentication methods we're going to implement very soon.

The azidentity supports many credential types https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#readme-credential-types, but I guess our list will probably be:

Client secret (oauth2) — this PR!
Workload identity
Managed Identity (?)
Client certificate (?)

I would adopt the azidentity naming convention and use "client secret" insted of "OAuth2".

Since we'll soon have 3+ authentication types, we should probably consider adding an "auth type" configuration setting for explicit selection. Inferring the authentication type from the option values can become tricky pretty quickly—also considering that we'll have event hub and storage account.

Having an auth_type option to check can probably simplify config validation and setup logic.

To wrap things up I suggest:

Move the authentication-related code into dedicated files, with a single entry point that takes the config and returns a client.
Add auth_type (for example, client_credentials, workload_identity, etc.)
Adopt the azidentity naming convention for credential types.
We probably need to list the permissions needed to read from the Event Hubs and write to the Storage Account. Ideally also add at least a link to some Microsoft docs that explains how to set up the service principal / app regististration—so we don't have to maintain it.

kaiyan-sheng · 2025-10-31T22:41:15Z

Since we'll soon have 3+ authentication types, we should probably consider adding an "auth type" configuration setting for explicit selection. Inferring the authentication type from the option values can become tricky pretty quickly—also considering that we'll have event hub and storage account.

@zmoog I totally forgot we talked about this in our meeting. Yes I will add the auth_type here. I think for backwards compatibility, I will just have connection_string as the default auth_type if auth_type is not specified.

kaiyan-sheng · 2025-10-31T22:55:14Z

@zmoog While making the changes, I realize we also have storage account auth type to consider. Should we allow customers to have different auth types for storage account and eventhub? Will the customer in any use case perfers that?

kaiyan-sheng · 2025-11-03T17:53:14Z

@zmoog and I talked offline and decided to only have one auth_type for both eventhub and storage account.

zmoog

We need to remove calls to the legacy SDKs from the processor v2 code path.

I have ideas to streamline the Validate() function in the config. It has become overly complex, but I'm also fine in dealing with this in a follow up PR.

docs/reference/filebeat/filebeat-input-azure-eventhub.md

x-pack/filebeat/input/azureeventhub/auth.go

x-pack/filebeat/input/azureeventhub/config.go

zmoog

LGTM

LJaganathan · 2025-12-17T12:57:24Z

Hello! @kaiyan-sheng @zmoog
"Azure Event Hub to Elasticsearch Integration Using Azure Client Credentials"

With respect to the integration between Azure Event Hub and Elasticsearch using Azure Active Directory authentication (Client ID and Client Secret), it is understood that this capability will be introduced in version 9.3. Could you please confirm the expected availability timeline for the upgraded version for customers?

Thanks,
-Jaganathan L

github-actions · 2025-12-17T16:32:51Z

@Mergifyio backport 8.19 9.1 9.2 9.3

mergify · 2025-12-17T16:33:08Z

backport 8.19 9.1 9.2 9.3

❌ No backport have been created

Details

#48142 [8.19](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 8.19 but encountered conflicts
#48143 [9.1](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 9.1 but encountered conflicts
#48144 [9.2](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 9.2 but encountered conflicts
Backport to branch 9.3 failed

GitHub error: Branch not found

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # docs/reference/filebeat/filebeat-input-azure-eventhub.md # x-pack/filebeat/input/azureeventhub/config.go

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # x-pack/filebeat/input/azureeventhub/config.go

…r eventhub filebeat input (#48142) * [Azure] Add client secret (Oauth2) support for eventhub filebeat input (#47256) (cherry picked from commit 7ffcd63)

… eventhub filebeat input (#48144)

… eventhub filebeat input (#48143)

kaiyan-sheng added 2 commits October 20, 2025 17:45

add oauth2 support

a0811eb

fix parsing connection string when using oauth2

c6ff76b

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 22, 2025

mergify bot assigned kaiyan-sheng Oct 22, 2025

support oauth2 with storage account

25a4dd1

kaiyan-sheng added the backport-skip Skip notification from the automated backport with mergify label Oct 28, 2025

kaiyan-sheng added 2 commits October 28, 2025 17:56

add tests

ba53a00

update doc

8f9f12b

kaiyan-sheng marked this pull request as ready for review October 29, 2025 23:22

kaiyan-sheng requested review from a team as code owners October 29, 2025 23:22

kaiyan-sheng requested a review from zmoog October 29, 2025 23:22

colleenmcginnis reviewed Oct 30, 2025

View reviewed changes

kaiyan-sheng added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 30, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 30, 2025

zmoog reviewed Oct 30, 2025

View reviewed changes

update doc

147f4f1

kaiyan-sheng changed the title ~~[Azure] Add Oauth2 support for eventhub filebeat input~~ [Azure] Add client secret (Oauth2) support for eventhub filebeat input Nov 3, 2025

kaiyan-sheng marked this pull request as draft November 4, 2025 01:14

kaiyan-sheng added 3 commits November 3, 2025 22:48

add auth_type

1c80794

fix unit tests

17d85a3

add more tests for the config options

2d1f3d9

kaiyan-sheng added 3 commits December 10, 2025 17:24

support diff domains for the storage account url

55ce7e8

remove config.Validate()

3ee1971

add extra line back

5602b8c

kaiyan-sheng requested a review from zmoog December 11, 2025 06:08

zmoog reviewed Dec 15, 2025

View reviewed changes

kaiyan-sheng added 2 commits December 15, 2025 23:33

remove using getAzureEnvironment

8728a59

refactor Validate() func with sub functions

a668c0f

zmoog approved these changes Dec 16, 2025

View reviewed changes

Merge branch 'main' into oauth2_azure

ba1d273

kaiyan-sheng enabled auto-merge (squash) December 16, 2025 15:19

kaiyan-sheng merged commit 7ffcd63 into main Dec 16, 2025
25 checks passed

kaiyan-sheng deleted the oauth2_azure branch December 16, 2025 16:35

kaiyan-sheng mentioned this pull request Dec 16, 2025

Add Oauth2 support for Azure EventHub integration elastic/integrations#16586

Closed

kaiyan-sheng added backport-active-all Automated backport with mergify to all the active branches and removed backport-skip Skip notification from the automated backport with mergify labels Dec 17, 2025

mergify bot pushed a commit that referenced this pull request Dec 17, 2025

[Azure] Add client secret (Oauth2) support for eventhub filebeat input (

18155b4

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # x-pack/filebeat/input/azureeventhub/config.go

mergify bot pushed a commit that referenced this pull request Dec 17, 2025

[Azure] Add client secret (Oauth2) support for eventhub filebeat input (

8974c0e

#47256) (cherry picked from commit 7ffcd63) # Conflicts: # x-pack/filebeat/input/azureeventhub/config.go

kaiyan-sheng pushed a commit that referenced this pull request Dec 19, 2025

[9.2](backport #47256) [Azure] Add client secret (Oauth2) support for…

aa03bec

… eventhub filebeat input (#48144)

kaiyan-sheng pushed a commit that referenced this pull request Dec 19, 2025

[9.1](backport #47256) [Azure] Add client secret (Oauth2) support for…

3d7194f

… eventhub filebeat input (#48143)

This was referenced Jan 12, 2026

[Microsoft Defender XDR]: (m365_defender) make OAuth2 endpoint params configurable elastic/integrations#16134

Open

[Microsoft Defender for Cloud]: (microsoft_defender_cloud) make OAuth2 endpoint params configurable elastic/integrations#16135

Open

a03nikki mentioned this pull request Jan 20, 2026

[Filebeat][azure-eventhub] Add support for Oauth2 extra parameters #48479

Open

kaiyan-sheng mentioned this pull request Jan 21, 2026

[Azure] Add client secret (Oauth2) support elastic/integrations#17030

Closed

zmoog mentioned this pull request Feb 3, 2026

[azure-eventhubs] expand authentication options #48679

Open

kaiyan-sheng mentioned this pull request Feb 5, 2026

Add Client Secret support into azure event hub integrations elastic/integrations#17245

Merged

17 tasks

Conversation

kaiyan-sheng commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

OAuth2 specific Configuration Parameters (auth_type=client_secret)

Checklist

Disruptive User Impact

How to test this PR locally

When no connection_string is specified and no auth_type is specified:

testing backwards compatibility:

testing with oauth2 for both eventhub and SA:

Screenshots

Logs

Uh oh!

github-actions bot commented Oct 22, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Oct 22, 2025

Uh oh!

github-actions bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

colleenmcginnis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 30, 2025

Uh oh!

kaiyan-sheng commented Oct 30, 2025

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng commented Oct 31, 2025

Uh oh!

kaiyan-sheng commented Oct 31, 2025

Uh oh!

kaiyan-sheng commented Nov 3, 2025

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LJaganathan commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

mergify bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ No backport have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kaiyan-sheng commented Oct 22, 2025 •

edited

Loading

OAuth2 specific Configuration Parameters (`auth_type`=`client_secret`)

When no `connection_string` is specified and no `auth_type` is specified:

github-actions bot commented Oct 22, 2025 •

edited

Loading

LJaganathan commented Dec 17, 2025 •

edited

Loading

mergify bot commented Dec 17, 2025 •

edited

Loading