Support for wildcards and override option for dot_expander processor#74601
Merged
felixbarny merged 9 commits intoelastic:masterfrom Jul 8, 2021
Merged
Conversation
Collaborator
|
Pinging @elastic/es-core-features (Team:Core/Features) |
felixbarny
commented
Jun 28, 2021
danhermann
suggested changes
Jun 28, 2021
...s/ingest-common/src/test/java/org/elasticsearch/ingest/common/DotExpanderProcessorTests.java
Show resolved
Hide resolved
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
Contributor
|
Does this address this issue ? #36950 |
Member
Author
|
Yep, it does. |
Member
Author
|
@danhermann I have implemented your suggestions. Could you have another look? |
Contributor
|
@elasticmachine run elasticsearch-ci/docs |
danhermann
approved these changes
Jul 7, 2021
Contributor
danhermann
left a comment
There was a problem hiding this comment.
Some minor suggestions below. No need for another round of review if they can be incorporated without issue.
modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/DotExpanderProcessor.java
Show resolved
Hide resolved
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
a1d46d6 to
aa2a673
Compare
Contributor
|
Thanks, LGTM. 👍 |
elasticsearchmachine
pushed a commit
to elasticsearchmachine/elasticsearch
that referenced
this pull request
Jul 8, 2021
Collaborator
💚 Backport successful
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the ability to
overrideconflicting properties instead of converting them to an array.Also, adds support for specifying a wildcard
*to apply the dot extension to every property.Background: This is important for creating a parsing pipeline for ECS JSON logs.
Due to human readability and the way some logging frameworks work (based on adding chars to a stream rather than maintaining a dictionary for JSON objects), ECS loggers use dotted field names and nested objects interchangeably (
foo.bar: baz,foo: { bar: baz }). As not all fields are known upfront, we need to dedot all fields.Good news is that we don't have to do that recursively, as a path is either fully dotted or fully nested. I.e. we don't have to consider cases like
foo: { bar.baz: qux }.The
overrideproperty is important for this use case:data_stream: { dataset: foo }, data_stream.dataset: bar. Expandingdata_streamwould lead todata_stream: { dataset: [foo, bar] }but this is not expected to be an array field. Instead, the dotted field should always override the nested field.