Add log.flags and object metadata to aws-s3 input events#26267
Merged
andrewkroh merged 3 commits intoelastic:masterfrom Jun 14, 2021
Merged
Add log.flags and object metadata to aws-s3 input events#26267andrewkroh merged 3 commits intoelastic:masterfrom
andrewkroh merged 3 commits intoelastic:masterfrom
Conversation
Contributor
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
e1a50d5 to
d0e7758
Compare
Contributor
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
d0e7758 to
4618a93
Compare
leehinman
reviewed
Jun 11, 2021
Contributor
leehinman
left a comment
There was a problem hiding this comment.
What do you think about adding the meta data to the s3Info struct?
The meta data feels like a good fit with the other data in s3Info, and we already passing an s3Info into all the functions so it would shorten the argument list.
Contributor
|
This pull request is now in conflicts. Could you fix it? 🙏 |
This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like `multiline` and `truncated` to indicate how the data was processed. This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback.
c414773 to
e8a0f65
Compare
Member
Author
@leehinman Good idea. Looks better after that change. |
leehinman
approved these changes
Jun 14, 2021
mergify bot
pushed a commit
that referenced
this pull request
Jun 14, 2021
* Add log.flags and object metadata to aws-s3 input events This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like `multiline` and `truncated` to indicate how the data was processed. This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback. * Pass metadata using s3Info struct to avoid adding new func params (cherry picked from commit c25fca8)
andrewkroh
added a commit
that referenced
this pull request
Jun 14, 2021
…6298) * Add log.flags and object metadata to aws-s3 input events This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like `multiline` and `truncated` to indicate how the data was processed. This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback. * Pass metadata using s3Info struct to avoid adding new func params (cherry picked from commit c25fca8) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
michalpristas
pushed a commit
to michalpristas/beats
that referenced
this pull request
Jun 17, 2021
* Add log.flags and object metadata to aws-s3 input events This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like `multiline` and `truncated` to indicate how the data was processed. This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback. * Pass metadata using s3Info struct to avoid adding new func params
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This adds the
log.flagsfield created by the line readers to aws-s3 events.log.flagscontains metadata likemultilineandtruncatedto indicate how the data was processed.This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the
Last-Modifiedtimestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback.Why is it important?
It makes the S3 log reader behave more similar to the log reader. And having access to the S3 headers gives users a bit more flexibility in their use-cases.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.