Add MySQL query metricset with lightweight module and SQL helper#18955
Merged
sayden merged 24 commits intoelastic:masterfrom Jul 13, 2020
Merged
Add MySQL query metricset with lightweight module and SQL helper#18955sayden merged 24 commits intoelastic:masterfrom
sayden merged 24 commits intoelastic:masterfrom
Conversation
Contributor
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
e3661af to
45e85bd
Compare
b14cf5f to
c34a9fd
Compare
Contributor
Author
|
I'm thinking to remove the Having it as a lightweight module, it is easy to include it later. |
Contributor
|
Pinging @elastic/integrations-services (Team:Services) |
ChrsMark
reviewed
Jul 7, 2020
Member
ChrsMark
left a comment
There was a problem hiding this comment.
Nice feature! I left some comments but in general looks good!
876ab17 to
41941aa
Compare
jsoriano
previously requested changes
Jul 7, 2020
Member
jsoriano
left a comment
There was a problem hiding this comment.
This is looking good, I think that is a good idea to have generic metricsets on the specific modules to have documentation together and a more consistent experience for users using an only db.
Changelog entries will be needed, there are many things here we should mention:
- New query metricset in the mysql module.
- New performance metricset in the mysql module.
- Add support for booleans in the generic sql module.
- Null values are not reported in the generic sql module.
Maybe for the developer changelog:
- New sql helper for metricbeat modules.
Contributor
Author
|
Very good comments indeed. Thanks! Will take a look |
kaiyan-sheng
reviewed
Jul 7, 2020
kaiyan-sheng
reviewed
Jul 7, 2020
ChrsMark
reviewed
Jul 8, 2020
jsoriano
reviewed
Jul 9, 2020
Member
jsoriano
left a comment
There was a problem hiding this comment.
Please try to regenerate the data.json files after adding the host parser and the replace_underscores option to the manifest.
bmorelli25
reviewed
Jul 13, 2020
metricbeat/docs/modules/sql.asciidoc
Outdated
| driver: "postgres" | ||
| sql_query: "SELECT * FROM pg_catalog.pg_tables pt WHERE schemaname ='pg_catalog'" | ||
| sql_response_format: table | ||
| --- |
Member
There was a problem hiding this comment.
This should fix the doc build
Suggested change
| --- | |
| ---- |
Member
There was a problem hiding this comment.
Oh but this is generated. I don't know where this is is coming from. @dedemorton probably will
Contributor
Author
There was a problem hiding this comment.
Oh please, I cannot believe it 😄
Contributor
Author
There was a problem hiding this comment.
Ok, pushed and fixed. Thank you very much 😃
sayden
added a commit
to sayden/beats
that referenced
this pull request
Jul 13, 2020
…stic#18955) (cherry picked from commit d0f7058)
Merged
5 tasks
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 14, 2020
* upstream/master: (25 commits) [Elastic Agent] Send checkin payload to Fleet (elastic#19857) [Ingest Manager] Fixed tests across agent elastic#19877 [Ingest Manager] Fix serialization test elastic#19876 Fix service start type mapping in windows/service metricset (elastic#19551) ci: Change comment trigger detection method (elastic#19827) Add 21 autogenerated filesets from rsa2elk devices (elastic#19713) [Ingest Manager] Agent config cleanup (elastic#19848) libbeat/publisher/pipeline: fix data races (elastic#19821) Update monitoring-internal-collection.asciidoc (elastic#19422) (elastic#19697) [Elastic Agent] Trust exchange endpoint must bind to 127.0.0.1 (elastic#19861) Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159) Add azure billing metricset (elastic#19207) Add support for appinsights in the metricbeat azure module (elastic#18940) Add MySQL query metricset with lightweight module and SQL helper (elastic#18955) [Ingest Manager] Refuse invalid stream values in configuration (elastic#19587) Do not use vendor during integration tests (elastic#19839) LIBBEAT: Enhancement Convert dissected values from String to other basic data types and IP (elastic#18683) [Elastic Agent] Remove support for "logs" and only support logfile (elastic#19761) [CI] support windows-2012 (elastic#19773) Do not update go.mod during packaging and testing (elastic#19823) ...
This was referenced Jul 22, 2020
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a SQL helper (at Metricbeat level) with logic to execute SQL queries and return their results. The module can use this abstraction layer to simply pass in the SQL driver and the queries they need to execute to later create Beats events.
This PR introduce:
sqlmodule.sqlmodule.queryMetricset in MySQL module which uses the SQL helper. This Metricset is to use it as a Lightweight "parent" module.performancemetricset using thequerymetricset as Lightweight parent module. Use it as a guidance about how to make it work.overviewmetricset with a bunch of key metrics, also using Lightweight modules. This is the metricset to remove of this PR.sqlmodule was moved to thesqlhelper.Why is it important?
The way the
sqlwas working was a bit limited compared to the underlying power it has. Moving it one layer up from the modules, allows the use of its logic by any module while maintaining each module functionality regarding specific DSN or security, for example, instead of introducing all that logic into thesqlmodule.The way it works now also allows to create metricsets on CockroachDB module using lightweight modules.
Checklist
[ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Related issues
Fixes: #18898
Implements: #15048