I did some testing with some nginx access logs. These are the steps that I followed

Setup a Elastic cluster with fleet and elastic-agent running in kubernetes. Since I have automation for this use case
I created a sample app running in Kubernetes that writes a file /var/log/nginx-logs in each of the kubernetes nodes
I made some changes to the nginx integration to add the following settings to the manifest packages/nginx/data_stream/access/manifest.yml

elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true

I have also added an entry to the changelog to be able to install the integration locally (via elastic-package and a local package registry) with a custom version and made the required changes to the manifest to point to this new version.

I installed the customized integration in fleet and added to my agent policy
I created this custom pipeline logs-nginx.access@custom that is called at the end of the default pipeline logs-nginx.access-1.12.1 (where 1.12.1 is the new version of the integration)

{
    "description": "My optional pipeline description",
    "processors": [
        {
            "set": {
                "description": "My optional processor description",
                "field": "my-long-field",
                "value": 10
            }
        },
        {
            "reroute": {
                "dataset": "test",
                "namespace": "test"
            }
        }
    ]
}

here the field is only to verify that the custom pipeline has been correctly called.

I created a data view over the datastream logs-test-test and checked the data in Discover.

Data is correctly rerouted from one datastream to the other one. There are no failures due to permissions (like in the linked issue without the extra settings).

Allow routing for input packages #5989

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions