Add support to set the document ID in the filebeat json reader#5844
Add support to set the document ID in the filebeat json reader#5844urso merged 4 commits intoelastic:masterfrom
Conversation
b9087d1 to
1bc2f1e
Compare
ruflin
left a comment
There was a problem hiding this comment.
I like this addition. I think we should require id to be a string as otherwise this could have some unwanted side effects.
There was a problem hiding this comment.
What happen if id is not a string?
There was a problem hiding this comment.
Adde a test for "id" not being a string.
Reading ID is similar to reading "MessageKey". If it's missing we return an empty string.
|
@urso After all the renaming, this needs a rebase. |
1bc2f1e to
24799c6
Compare
|
@ruflin rebased. Also update PR to enforce 'id' to be a string. |
|
This seems to break quite a few tests. Other note: We will also need to add a doc entry for this. Part of this PR or follow up issue? |
Add support to configure a key for setting the document ID in the harvester JSON settings. The ID will be store in the events Meta["id"] for the output to pick up. With elastic#5811 will the elasticsearch output is the Meta["id"] field to set the document its ID (uses op_type="create" to count duplicate inserts of same ID). For other output type, the document ID will be forwarded via `@metadata.id`.
2374406 to
232f6db
Compare
The PR already modifies the filebeat json config docs |
ruflin
left a comment
There was a problem hiding this comment.
Overall code LGTM, would be nice to have this documented to. I think it's also important to know, if someone chooses this option, it will mean a document will be overwritten in ES if the same id shows up again?
|
There is some minimal doc update in this PR. But there are some other places throughout the docs where we want to document this feature. Labeling as needs_doc and creating doc issue to follow up. |
Requires: #5811
This PR adds support to configure a key for setting the document ID in the harvester JSON settings. The ID will be store in the events Meta["id"] for the output to pick up. With #5811 will the elasticsearch output is the Meta["id"] field to set the document its ID (uses op_type="create" to count duplicate inserts of same ID). For other output type, the document ID will be forwarded via
@metadata.id.