Can we make sure that this processor is the same in all integrations?

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

🤖 GitHub comments

🌐 Coverage report

Can we make sure that this processor is the same in all integrations?

i think it's kinda crazy each integration repeats these processors anyway. is there any reason not to move this into a higher level final pipeline?

@ruflin i think we talked about this at one point?

i think it's kinda crazy each integration repeats these processors anyway. is there any reason not to move this into a higher level final pipeline?

I see a few problems:

• In the Azure integration I want to preserve event.original
• In the AWS Integration I don't want to preserve event.original

Some sort of tag must be send to the final pipeline, so it actually knows when to set it and when not.

Problem: Custom ingest pipelines. what if those heavily modify the message field, or even drop it? Then the final pipeline won't be able to create the event.original, since message is missing, or something else happened.

@philippkahr, the event.original field is preserved or removed from the document according to the presence of the preserve_original_event tag. This tag is a data stream-level option, so each individual data stream can pick its value.

The change in this PR handles an edge case that can happen the pipeline receives an event with both message and event.original fields. It is performed early in the pipeline and aims to have the event.original field only in the document, which will be kept or dropped according to the preserve_original_event tag.

(IIRC) This change was first introduced when Logstash changed its default behavior a few months ago and added event.original.

Package azure - 1.5.9 containing this change is available at https://epr.elastic.co/search?package=azure

Also seeing error.message: field [event.original] already exists rename in the azure application gateway logs with preserve original log disabled.

Also seeing error.message: field [event.original] already exists rename in the azure application gateway logs with preserve original log disabled.

Yep, this can happen at the beginning of the ingest pipeline execution if the document has both message and event.original.

@mbudge, is it still happening with integration version 1.5.9? Can you provide a sample log causing the error if it does happen?

We've upgraded to 8.6.2 but the latest azure firewall/application gateway integration is 1.5.8.

As soon as 1.5.9 is available for upgrade we'll test it.

Just upgraded to 1.5.9 and the original event issue is fixed in data_stream.dataset: azure.firewall_logs

The firewall integration has a grok parsing error, I'll share with support.

We're still getting the error "field [event.original] already exists rename" this dataset in v1.5.9

data_stream.dataset: azure.application_gateway

The firewall integration has a grok parsing error, I'll share with support.

Feel free to also open a public GitHub issue with the failing log event. Please redact any PII or account-identifiable information before sharing it.

We're still getting the error "field [event.original] already exists rename" in data_stream.dataset: azure.application_gateway

You're right!

Fixing this with #5361

The firewall integration has a grok parsing error, I'll share with support.

We added unsupported message patterns with #5382. Thank you for sharing a sample log.

You can upgrade to integration version 1.5.11 when the automation bot confirms.