Skip to content

Add nats module to Metricbeat#9825

Merged
ruflin merged 21 commits intoelastic:masterfrom
nfvri:master
Jan 15, 2019
Merged

Add nats module to Metricbeat#9825
ruflin merged 21 commits intoelastic:masterfrom
nfvri:master

Conversation

@ChrsMark
Copy link
Copy Markdown
Member

This PR adds nats on top of Metricbeat as a module as proposed in #9674.

Tests need to be added, but an early review would be of benefit here.

cc: @ruflin, @exekias

Co-Authored-by: Stamatis Katsaounis katsaouniss@gmail.com, @skatsaounis
Co-Authored-By: Michael Katsoulis michaelkatsoulis88@gmail.com, @MichaelKatsoulis

@ChrsMark ChrsMark requested a review from a team as a code owner December 28, 2018 14:05
@elasticmachine
Copy link
Copy Markdown
Contributor

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@ChrsMark ChrsMark changed the title Initial commit nats module Adding nats module to Metricbeat Dec 29, 2018
@sayden
Copy link
Copy Markdown
Contributor

sayden commented Jan 2, 2019

Hi @ChrsMark

Thank you for contributing to this! Few things are needed before merging this module, let's start with field count and naming conventions:

  • The field count is way too high. While we understand that it's easy to just "parse all", later a detailed description of each field must be written in the docs and no field has any description. We try to maintain just the fields that are key to understand the health of a system and detail them well in the docs. This way the index is also smaller in size and the dashboards aren't huge. If it helps, try to think what will you place in a Kibana dashboard and remove everything else. Less is more 😉
  • Fields must follow this naming conventions. For example, out_bytes must be an object called out with a sub-object called bytes

Let's start with those 2 things and keep pushing from there.

@sayden sayden self-assigned this Jan 2, 2019
@sayden sayden added module Metricbeat Metricbeat labels Jan 2, 2019
@urso urso removed the request for review from a team January 2, 2019 15:20
@ChrsMark ChrsMark requested review from a team as code owners January 4, 2019 16:09
@ruflin ruflin added the Team:Integrations Label for the Integrations team label Jan 7, 2019
@ChrsMark ChrsMark requested a review from a team as a code owner January 7, 2019 09:10
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 7, 2019

Any chance we could split up this PR in 4 PR's? First PR introducing the module and 1 metricset and then 3 follow up PR's with 1 metricset each. This would make the review easier and should get each PR in quicker.

Could you also add a changelog entry?

@ChrsMark
Copy link
Copy Markdown
Member Author

ChrsMark commented Jan 7, 2019

Hey @ruflin,

we will go along with your proposal on splitting the PR in 4 smaller.
Is there any standard process you use to do such things, or we can just open the 4 new and abandon this one?

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 7, 2019

No standard way on doing this, whatever works best for you. You could also keep this one as the "foundation" PR for the module + 1 metricset and open 3 new ones.

@urso urso removed request for a team January 7, 2019 21:03
@ChrsMark ChrsMark changed the title Adding nats module to Metricbeat Add nats module to Metricbeat Jan 7, 2019
@ChrsMark
Copy link
Copy Markdown
Member Author

ChrsMark commented Jan 8, 2019

Hi @ruflin, the PR seems to be ready for review.

So far:

  1. @sayden's comments have been addressed
  2. this PR refers to the module and 1 metricset. 3 PRs will follow after this one is accepted to avoid having similiar comments/requested-changes

Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good already. I especially like that you added most of the tests. It would be nice if you could also add a system tests that runs the beat and checks if all fields are documented. See here for an example: https://github.com/elastic/beats/blob/master/metricbeat/tests/system/test_haproxy.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kind of surprised it's reported like this. I assume ES does the conversion automatically?

@ChrsMark
Copy link
Copy Markdown
Member Author

@ruflin we are done with the changes, could you check again?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somehow duplicated here but we can fix this later.

@sayden
Copy link
Copy Markdown
Contributor

sayden commented Jan 11, 2019

I know I'm coming a bit late but I suggest to change the name of the metricset from varz to general or stats as you can read in the description here. What I want is to have a meaningful metricset name instead of the path of the endpoint you request to fetch the data. This will also protect the module from a potential change in the path of the endpoint.

What do you think @ruflin ?

Copy link
Copy Markdown
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some easy changes. In general looks very good. I have omitted every single change in the naming conventions but I have left you some ideas that should guide you with the rest of the names. Feel free to ask as much as you want. Good work 😉

@ChrsMark
Copy link
Copy Markdown
Member Author

ChrsMark commented Jan 11, 2019

@sayden, @ruflin great loop so far! Apart from the minors you guys mentioned, should we check the change from varz to stats regarding the name of the metricset? To my mind stats is more meaningful, and we should spend the time on it, making it more robust and future proof.

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 11, 2019

I'm on board with renaming the metricset. The goal is to make the end document as easy as possible to consume.

skatsaounis and others added 18 commits January 15, 2019 10:02
Signed-off-by: Stamatis Katsaounis <katsaouniss@gmail.com>
Signed-off-by: Stamatis Katsaounis <katsaouniss@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Stamatis Katsaounis <katsaouniss@gmail.com>
Co-Authored-By: Michael Katsoulis <michaelkatsoulis88@gmail.com>
Signed-off-by: MichaelKatsoulis <michaelkatsoulis88@gmail.com>
Co-Authored-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: MichaelKatsoulis <michaelkatsoulis88@gmail.com>

Co-Authored-by: ChrisMark <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Michael Katsoulis <michaelkatsoulis88@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: MichaelKatsoulis <michaelkatsoulis88@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
Signed-off-by: Michael Katsoulis <michaelkatsoulis88@gmail.com>
Signed-off-by: Chris <chrismarkou92@gmail.com>
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 15, 2019

@ChrsMark To get CI green, I think you have to run make update again.

Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM. Now let's get CI green :-)

Copy link
Copy Markdown
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, let's wait CI to get green

Signed-off-by: Chris <chrismarkou92@gmail.com>
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 15, 2019

jenkins, test this

@ChrsMark
Copy link
Copy Markdown
Member Author

Is this failure relevant @ruflin?

@ruflin ruflin merged commit aa7e600 into elastic:master Jan 15, 2019
@ruflin ruflin mentioned this pull request Jan 15, 2019
15 tasks
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 15, 2019

@ChrsMark It's not, just merged it. Thanks a lot for your contribution.

I also created #10071 as a follow up issue to track the completion of the module: #10071

@ChrsMark
Copy link
Copy Markdown
Member Author

Cool @ruflin!
Follow up PRs (3) will come soon from @skatsaounis, @MichaelKatsoulis and me.

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 15, 2019

Great, looking forward to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Metricbeat Metricbeat module Team:Integrations Label for the Integrations team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants