Cherry-pick #23334 to 7.x: [Filebeat] Add postgresql csv fileset#24088
Merged
jsoriano merged 1 commit intoelastic:7.xfrom Feb 17, 2021
Merged
Cherry-pick #23334 to 7.x: [Filebeat] Add postgresql csv fileset#24088jsoriano merged 1 commit intoelastic:7.xfrom
jsoriano merged 1 commit intoelastic:7.xfrom
Conversation
Refactor PostgreSQL module to support logs in CSV format. The fileset has three pipelines now, first one is executed to parse the initial part of the lines, till it decides if the logs are plain text or CSV, once decided it invokes one of the other two pipelines, one specific for plain text logs and the other for CSV. Several test cases and documentation are added for CSV support. Additional fields are available now, and some others have been renamed to represent more accurately the values they store. Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> (cherry picked from commit bb78931)
Contributor
|
Pinging @elastic/integrations (Team:Integrations) |
Contributor
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
kaiyan-sheng
approved these changes
Feb 17, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #23334 to 7.x branch. Original message:
As a configuration option, create another pipeline inside the module to
parse CSV log files.
Although it's necessary to configure the database to emit such logs,
there is some advantages: configured properly, some events like
statement timeoutandlock timeoutwill display the query in thesame event, opposed to multiple lines, one with the error message and
other with statement.
What does this PR do?
This PR creates a new fileset/pipeline inside PostgreSQL module. This new fileset parses and create events based on CSV log files from postgresql.
Why is it important?
PostgreSQL logs weren't designed to be parsed, so, without very complicated logic, it's not possible to encapsulate all information in just one event. One example of that is
statement timeout:Using the CSV log, those 2 lines become 1:
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist