Description
It's common for the message in a log event to actually be a JSON itself. Specifically, there are shippers that produce ECS-JSON.
We would like to handle this automatically through the normalize_for_stream ingest processor. The idea is to add an additional step, so that it now does the following:
- If it's OTel data: just use as is
- If it's not OTel:
- apply a cheap check whether
message is a JSON-encoded string (e.g. if it starts with { and ends with })
- If yes:
- Parse the
message as JSON*
- Apply a cheap check whether the resulting object is ECS (e.g. if contains a
@timestamp key)
- If it's ECS: merge the resulting object back into the root of the document*
- If not: add the resulting object as is as the value of the
body.structured field
- Proceed with namespacing/normalization as before
* JSON parsing and merge should behave exactly exactly as defined in logs@json-pipeline
@dakrone is it a problem to have dependencies between one processor and another? More specifically, is it possible for the NormalizeForStreamProcessor to use JsonProcessor#apply?
Description
It's common for the message in a log event to actually be a JSON itself. Specifically, there are shippers that produce ECS-JSON.
We would like to handle this automatically through the
normalize_for_streamingest processor. The idea is to add an additional step, so that it now does the following:messageis a JSON-encoded string (e.g. if it starts with{and ends with})messageas JSON*@timestampkey)body.structuredfield* JSON parsing and merge should behave exactly exactly as defined in
logs@json-pipeline@dakrone is it a problem to have dependencies between one processor and another? More specifically, is it possible for the
NormalizeForStreamProcessorto useJsonProcessor#apply?