Skip to content

[azure logs] Make routing.category field available to pipelines#16610

Merged
zmoog merged 5 commits intomainfrom
zmoog/ktlo/azure/support-routing-overrides
Dec 23, 2025
Merged

[azure logs] Make routing.category field available to pipelines#16610
zmoog merged 5 commits intomainfrom
zmoog/ktlo/azure/support-routing-overrides

Conversation

@zmoog
Copy link
Copy Markdown
Contributor

@zmoog zmoog commented Dec 17, 2025 •
edited
Loading

Proposed commit message

Support for custom log event routing in the Azure Logs v2 integration.

The Azure Logs v2 default pipeline now extracts the routing.category field from the log event and makes it available to custom pipelines, allowing users to apply custom routing logic.

For example, SQLSecurityAuditEvents would normally be routed to the azure.platformlogs dataset because Synapse log events are not yet supported natively.

By creating the logs-azure.events@custom pipeline with the following content:

PUT _ingest/pipeline/logs-azure.events@custom
{
  "processors": [
    {
      "reroute": {
        "if": "ctx?.routing?.category == \"SQLSecurityAuditEvents\"",
        "dataset": "azure.synapse"
      }
    }
  ]
}

Users can now route these events to a dedicated azure.synapse dataset, or an existing dataset, like azure.activitylogs.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Define the logs-azure.events@custom with custom routing logic:

PUT _ingest/pipeline/logs-azure.events@custom
{
  "processors": [
    {
      "reroute": {
        "if": "ctx?.routing?.category == \"SQLSecurityAuditEvents\"",
        "dataset": "azure.synapse"
      }
    }
  ]
}

Send a sample log event:

POST logs-azure.events-default/_doc
{
  "@timestamp": "2025-12-17T07:15:26+01:00",
  "message": """{"category":"SQLSecurityAuditEvents"}"""
}

Search for the results:

POST logs-azure.synapse-default/_search
Results 
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-azure.synapse-default-2025.12.18-000001",
        "_id": "AZswXwsnopg1bPjToWtL",
        "_score": 1,
        "_source": {
          "@timestamp": "2025-12-17T06:15:26.000Z",
          "data_stream": {
            "dataset": "azure.synapse",
            "namespace": "default",
            "type": "logs"
          },
          "ecs": {
            "version": "8.11.0"
          },
          "event": {
            "dataset": "azure.synapse",
            "kind": "event"
          },
          "message": """{"category":"SQLSecurityAuditEvents"}""",
          "routing": {
            "category": "SQLSecurityAuditEvents"
          }
        }
      }
    ]
  }
}

Related issues

Screenshots

Documentation:

CleanShot 2025-12-18 at 08 57 40@2x

@zmoog
Make routing.category field available to pipelines
4f2f453 
Now the `logs-azure.events@custom` pipeline can use the
`routing.category` field to optionally override the routing.

Here's a simple example:

```
PUT _ingest/pipeline/logs-azure.events@custom
{
    "processors": [
      {
        "set": {
          "field": "event.dataset",
          "value": "azure.whatever",
          "if": "ctx.routing?.category == \"Whatever\""
        }
      }
    ]
}
```
@zmoog zmoog self-assigned this Dec 17, 2025
@zmoog zmoog added Integration:azure Azure Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Dec 17, 2025
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 17, 2025 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kaiyan-sheng
kaiyan-sheng reviewed Dec 18, 2025
View reviewed changes
packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml
description: "Parses the message field as JSON and stores it in a temporary field to identify the event dataset."

- rename:
field: tmp_json.category
Copy link
Copy Markdown

@kaiyan-sheng kaiyan-sheng Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to consider CategoryValue, category, Catetory?

Copy link
Copy Markdown
Contributor Author

@zmoog zmoog Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this pipeline is the entry point for all integrations, so it must take into account all the existing variation of fields containing the category name.

We currently cover:

  • category
  • Category
  • CategoryValue

@zmoog zmoog added Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Dec 18, 2025
@zmoog zmoog marked this pull request as ready for review December 18, 2025 07:59
@zmoog zmoog requested review from a team as code owners December 18, 2025 07:59
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 18, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@zmoog
Copy link
Copy Markdown
Contributor Author

zmoog commented Dec 18, 2025

Note for reviewers: I think the routing.category idea has legs, but the implementation feels a bit clunky right now. I'd love some feedback or suggestions on how to clean it up! :man-bowing:

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 23, 2025

💚 Build Succeeded

History

cc @zmoog

@zmoog zmoog merged commit 7b73180 into main Dec 23, 2025
8 checks passed
@zmoog zmoog deleted the zmoog/ktlo/azure/support-routing-overrides branch December 23, 2025 15:50
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 23, 2025

Package azure - 1.33.0 containing this change is available at https://epr.elastic.co/package/azure/1.33.0/

@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jan 8, 2026
@zmoog zmoog mentioned this pull request Jan 22, 2026
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrisberkhout chrisberkhout chrisberkhout approved these changes

@gizas gizas gizas approved these changes

@ishleenk17 ishleenk17 ishleenk17 approved these changes

+1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@zmoog zmoog

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:azure Azure Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[azure logs] Support routing overrides in Azure Logs v2

7 participants

@zmoog @elasticmachine @chrisberkhout @gizas @kaiyan-sheng @ishleenk17 @andrewkroh