Skip to content

IBM MQ Metricbeat module#8870

Closed
felix-lessoer wants to merge 40 commits intoelastic:masterfrom
felix-lessoer:mqMetricbeat
Closed

IBM MQ Metricbeat module#8870
felix-lessoer wants to merge 40 commits intoelastic:masterfrom
felix-lessoer:mqMetricbeat

Conversation

@felix-lessoer
Copy link
Copy Markdown
Contributor

This is the metricbeat module for IBM MQ. Which will be the transformation of this standalone beat

@ruflin is helping to integrate this as this is the first x-pack module for metricbeat

@ruflin ruflin added module review Metricbeat Metricbeat in progress Pull request is currently in progress. labels Nov 1, 2018
@ruflin ruflin self-requested a review November 1, 2018 09:56
@felix-lessoer
Copy link
Copy Markdown
Contributor Author

felix-lessoer commented Nov 6, 2018

The module is now able to collect the MQ metrics.
A few things are open:

1.) In the module each metricseat is collected in parallel. This is not possible with the MQ lib at the moment. @ruflin Is there a way to collect the metricsets sequentially without putting everything in one fetch metricset?
If not I need to change the way I connect to MQ. Each metricset needs to connect to its own command / reply queue .

2.) To build metricbeat with ibmmq module it needs to have the MQ C library available locally under

for Linux

  • cgo !windows CFLAGS: -I/opt/mqm/inc -D_REENTRANT
  • cgo !windows LDFLAGS: -L/opt/mqm/lib64 -lmqm_r -Wl,-rpath=/opt/mqm/lib64 -Wl,-rpath=/usr/lib64

for Windows

  • cgo windows CFLAGS: -I"C:/Program Files/IBM/MQ/Tools/c/include"
  • cgo windows LDFLAGS: -L "C:/Program Files/IBM/MQ/bin64" -lmqm

This needs to be switched to dynamic linking in mqi.go

3.) The PubSub metricset needs to be added

4.) The imports are pointing to my repo. This needs to be switched when it will be merged into Elastic repo

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Nov 6, 2018

That all metricset fetches are made in parallel is by design. To run it sequentially it would require quite fundamental changes. I would definitively recommend that each metricset maintains it's own connection, so if one metricset is blocked it does not impact others. Are there some advantages to this?

@ruflin ruflin added Team:Integrations Label for the Integrations team and removed Team:Integrations Label for the Integrations team labels Nov 21, 2018
@felix-lessoer felix-lessoer requested a review from a team as a code owner May 10, 2019 08:30
@ruflin ruflin added the Team:Integrations Label for the Integrations team label May 10, 2019
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented May 10, 2019

Some notes on my end for this PR:

  • We need to add a docker file for testing and generate the some expected outputs
  • Perhaps split PR up per metricset
  • Check what the impact of the dependencies on the size of the binary is
  • Add libraries with govendor to make sure they are properly vendored and in the correct directory.

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented May 10, 2019

Here is the docker image that can be used for testing: https://hub.docker.com/r/ibmcom/mq/

@felix-lessoer
Copy link
Copy Markdown
Contributor Author

Here is the docker image that can be used for testing: https://hub.docker.com/r/ibmcom/mq/

You need to create it with a Queue Manager Name
MQ_QMGR_NAME - Set this to the name you want your Queue Manager to be created with.

This Queue Manager Name needs to be in the configuration.
The connection config (cc) for Docker needs to look something like this to get a result:

cc:
clientMode: true
mqServer: "DEV.ADMIN.SVRCONN/TCP/127.0.0.1 (1416)"
userId: "admin"
password: "passw0rd"

Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving some comments for whoever follows up with the fixes to not forget about things.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must be pulled in as vendor directory to make the separation clear.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete the TODO in the docs

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@mtojek
Copy link
Copy Markdown
Contributor

mtojek commented Jan 2, 2020

I believe we can close this PR in favor of a simple light module implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

in progress Pull request is currently in progress. Metricbeat Metricbeat module review Team:Integrations Label for the Integrations team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants