Update PostgreSQL integration to support logs in CSV format#747
Update PostgreSQL integration to support logs in CSV format#747jsoriano merged 33 commits intoelastic:masterfrom
Conversation
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪 |
aba872a to
81d4bb2
Compare
81d4bb2 to
9d3ac0d
Compare
To avoid errors in empty dashboards like `Could not locate that index-pattern-field (id: event.duration)`.
|
Failing test is caused by elastic/beats#24359 |
|
Pinging @elastic/integrations (Team:Integrations) |
|
Opening for review, elastic/beats#24359 has been merged and should solve the failing test with next snapshots. |
|
/test |
mtojek
left a comment
There was a problem hiding this comment.
Nice improvement for PostgreSQL!
BTW You can rebase against master to run tests only for this integration (I pushed a fix yesterday).
| description: 'Name of the module this data is coming from. | ||
|
|
||
| If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module.' | ||
| example: apache |
There was a problem hiding this comment.
nit: I used to leave name, type and description in most cases (to shrink files), but you can leave all properties if you prefer.
There was a problem hiding this comment.
I copied directly from the ecs files in beats, but I will review it.
There was a problem hiding this comment.
I am leaving it as is, not sure what to remove.
|
While regenerating dashboards I have seen that some log lines are not being correctly parsed, taking a look. |
I hadn't configured PostgreSQL logs as Filebeat expects. There is an open issue about improving docs for this: elastic/beats#14337. More details about my investigation on this here: elastic/beats#20321 (comment) There are a couple of things we could improve to avoid these issues, but I'd like to have this merged before. @mtojek I think this would be ready for another review. Thanks! |
| required: true | ||
| type: keyword | ||
| ignore_above: 1024 | ||
| description: 'ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. |
There was a problem hiding this comment.
nit: do we need the quotes here for all descriptions?
There was a problem hiding this comment.
If they are multi-lined descriptions they need to be between quotes, or with yaml blocks started with | or |-.
I think elastic-package format fixed some of them, but I see we have all the options along these fields, so not sure which one to use, I would leave it as is by now.
mtojek
left a comment
There was a problem hiding this comment.
Ship it! Thank you for all adjustments.
What does this PR do?
Import latest changes in Beats. Including support for logs in CSV format (see #720, elastic/beats#23334).
packages/postgresql/_dev/build/docs/README.md.PACKAGES=postgresql mage ImportBeats.Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots