Updating Metricbeat stack modules to ECS by ycombinator · Pull Request #10350 · elastic/beats

ycombinator · 2019-01-26T15:29:52Z

This PR updates a few fields in the Elastic stack modules in Metricbeat, viz. elasticsearch, logstash, and kibana to their ECS names and types.

Based on discussions happening in #10218, but only scoped to Elastic stack modules.

elasticmachine · 2019-01-26T15:29:54Z

Pinging @elastic/stack-monitoring

ruflin

You also need to update the ecs-migration.yml file.

CHANGELOG.next.asciidoc

metricbeat/module/kibana/stats/_meta/fields.yml

metricbeat/module/logstash/node/_meta/data.json

ycombinator · 2019-01-28T15:39:55Z

@ruflin This is ready for another review pass. I addressed feedback from your previous review and left a couple of questions as well. Thanks!

webmat

Looking pretty good. Only need to resolve on the PID inconsistency. Also noticed a few service.version that haven't been migrated yet.

Once these are settled, this LGTM

metricbeat/module/elasticsearch/index_recovery/_meta/fields.yml

metricbeat/module/logstash/node/_meta/data.json

metricbeat/module/logstash/node/_meta/fields.yml

metricbeat/module/kibana/stats/_meta/data.json

metricbeat/module/kibana/status/_meta/data.json

metricbeat/module/logstash/node/_meta/data.json

ycombinator · 2019-01-28T23:53:43Z

@webmat @ruflin Addressed latest round of feedback. This is ready for your 👀 again, when you get a chance. Thanks!

ycombinator · 2019-01-29T02:16:55Z

jenkins, test this

dev-tools/ecs-migration.yml

ruflin

Overall LGTM. One minor comment.

I think we need to figure out what we do with the hostname fields. What we could do is get this PR in without the hostname fields and figure it out as a follow up.

metricbeat/module/logstash/node_stats/_meta/data.json

ycombinator · 2019-01-29T13:23:30Z

@ruflin Other than your second thoughts comments that require third thoughts from @webmat 😉 I've addressed all your other feedback.

webmat · 2019-01-29T15:47:24Z

@ruflin @ycombinator I think for both transport_address and hostname we could have them under service.*. This would neatly solve the problem of monitoring services from the outside vs locally.

ycombinator · 2019-01-29T16:01:29Z

@webmat said:

I think for both transport_address and hostname we could have them under service.*. This would neatly solve the problem of monitoring services from the outside vs locally.

Just to clarify, you're saying that:

in the case of Updating Metricbeat stack modules to ECS #10350 (comment), we should move the field to service.transport_address instead of kibana.stats.transport_address as this PR is doing currently, and
in the case of Updating Metricbeat stack modules to ECS #10350 (comment), we should move the field to service.hostname instead of host.hostname as this PR is doing currently.

webmat · 2019-01-29T17:27:16Z

Mostly yes, but I'll clarify one last thing 😆

I think kibana.stats.transport_address should be migrated to service.address

ycombinator · 2019-01-30T19:56:58Z

@webmat Took another shot at the language per your recommendations. Ready for your 👀 again, when you get a chance. Thanks!

webmat

Great! Thanks @ycombinator

LGTM

webmat · 2019-01-30T20:43:21Z

Of course I had to click too fast and not make this "Approved" 😆

…de_stats metricset

ycombinator added in progress Pull request is currently in progress. module Metricbeat Metricbeat v7.0.0 ecs Feature:Stack Monitoring breaking change labels Jan 26, 2019

ycombinator requested review from a team as code owners January 26, 2019 15:29

ycombinator mentioned this pull request Jan 26, 2019

Move Metricbeat module fields to ECS #10218

Closed

ycombinator added review and removed in progress Pull request is currently in progress. labels Jan 26, 2019

ycombinator requested review from ruflin and webmat January 26, 2019 15:54

ycombinator changed the title ~~[WIP] Updating Metricbeat stack modules to ECS~~ Updating Metricbeat stack modules to ECS Jan 26, 2019

ruflin requested changes Jan 28, 2019

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

metricbeat/module/kibana/stats/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/kibana/stats/_meta/fields.yml Outdated Show resolved Hide resolved

ruflin mentioned this pull request Jan 28, 2019

Transition Beats to ECS #8655

Closed

ycombinator commented Jan 28, 2019

View reviewed changes

metricbeat/module/logstash/node/_meta/data.json Outdated Show resolved Hide resolved

cachedout approved these changes Jan 28, 2019

View reviewed changes

webmat reviewed Jan 28, 2019

View reviewed changes

ruflin reviewed Jan 29, 2019

View reviewed changes

dev-tools/ecs-migration.yml Outdated Show resolved Hide resolved

ruflin reviewed Jan 29, 2019

View reviewed changes

metricbeat/module/logstash/node_stats/_meta/data.json Outdated Show resolved Hide resolved

webmat reviewed Jan 30, 2019

View reviewed changes

webmat approved these changes Jan 30, 2019

View reviewed changes

ycombinator added 22 commits January 30, 2019 19:47

Updating field names to ECS in field.yml and generated files

0eaf78a

Updating kibana module parsing code

98e368f

Updating logstash module parsing code

3527606

Adding CHANGELOG entries

f70619c

Fixing location of CHANGELOG entries

54ef869

Fixing up fields per ECS

91ae2ca

Fixing up kibana stats parsing and regenerating data.json file

c3d53ed

Kibana stats metricset changes

69a43ad

Updating Kibana status metricset

9f577dd

Updating Logstash metricsets

9c06874

Adding to ecs-migration.yml

313190e

Setting service.version for kibana metricsets

7ada215

Keeping only global process.pid for logstash/node metricset

341954b

Reverting transport_address change in kibana/stats metricset

8981925

Adding service.id for logstash/node metricset

54e4917

Adding host.hostname, service.id, and service.version for logstash/no…

d8ef445

…de_stats metricset

Review feedback

2cb76a5

Reverting changes leftover from testing

0c54748

Adding service.hostname to fields.common.yml

5f16418

Updating language

85bce07

Regenerating generated files

dfce436

Regenerating generated files

b389f7d

ycombinator merged commit ae83e46 into elastic:master Jan 31, 2019

ycombinator deleted the mb-stack-ecs branch December 25, 2019 11:15

matschaffer mentioned this pull request May 25, 2022

[Stack Monitoring] Logstash module logs continual error when monitoring multiple pipelines #31739

Closed

Kerry350 mentioned this pull request Jun 17, 2022

[Metricbeat] Stop continual error with multiple Logstash pipelines #31985

Merged

6 tasks

Conversation

ycombinator commented Jan 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jan 26, 2019

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ycombinator commented Jan 28, 2019

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ycombinator commented Jan 28, 2019

Uh oh!

ycombinator commented Jan 29, 2019

Uh oh!

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ycombinator commented Jan 29, 2019

Uh oh!

webmat commented Jan 29, 2019

Uh oh!

ycombinator commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webmat commented Jan 29, 2019

Uh oh!

ycombinator commented Jan 30, 2019

Uh oh!

webmat left a comment

Choose a reason for hiding this comment

Uh oh!

webmat commented Jan 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ycombinator commented Jan 26, 2019 •

edited

Loading

ycombinator commented Jan 29, 2019 •

edited

Loading