Skip to content

Convert Filebeat kafka.* to ECS#9297

Merged
webmat merged 4 commits intoelastic:masterfrom
webmat:ecs-kafka-fb
Dec 20, 2018
Merged

Convert Filebeat kafka.* to ECS#9297
webmat merged 4 commits intoelastic:masterfrom
webmat:ecs-kafka-fb

Conversation

@webmat
Copy link
Copy Markdown
Contributor

@webmat webmat commented Nov 29, 2018 •
edited
Loading

Caveats

  • Module used to keep original message in message, and cleaned up message
    in kafka.log.message. Keeping original message will be addressed elsewhere,
    and this PR now replaces message with the cleaned up message.

Renames

  • kafka.log.level => log.level
  • kafka.log.message => message

TODO

  • Alias renamed fields to their ECS counterpart
  • Document field migrations in ecs-migration.yml
  • Changelog
  • Final rebase

Noticed

  • This module doesn't have a sample log that populates kafka.log.trace.full

@ruflin ruflin mentioned this pull request Nov 29, 2018
Closed
@webmat webmat self-assigned this Nov 29, 2018
@webmat webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 29, 2018
@webmat webmat mentioned this pull request Dec 17, 2018
Closed
14 tasks
@webmat webmat force-pushed the ecs-kafka-fb branch 2 times, most recently from cafba24 to f30da0d Compare December 19, 2018 18:29
@webmat webmat requested a review from a team as a code owner December 19, 2018 18:29
@webmat webmat added the review label Dec 19, 2018
@webmat webmat changed the title WIP Convert Filebeat kafka.* to ECS Convert Filebeat kafka.* to ECS Dec 19, 2018
@webmat webmat removed the in progress Pull request is currently in progress. label Dec 19, 2018
ruflin
ruflin reviewed Dec 20, 2018
View reviewed changes
filebeat/module/kafka/log/ingest/pipeline.json Outdated
Copy link
Copy Markdown
Contributor

@ruflin ruflin Dec 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like one day we should have something related to Java class.

@felixbarny Is there a standardised way Java APM agent is storing this info?

ruflin
ruflin reviewed Dec 20, 2018
View reviewed changes
filebeat/module/kafka/log/test/state-change-1.1.0.log-expected.json Outdated
Copy link
Copy Markdown
Contributor

@ruflin ruflin Dec 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a log level we haven't really used yet. No change needed.

@webmat webmat force-pushed the ecs-kafka-fb branch 2 times, most recently from baf3209 to 0b77277 Compare December 20, 2018 16:12
Mathieu Martin added 4 commits December 20, 2018 14:26
Map 2 fields to ECS:
df67448 
- kafka.log.level => log.level
- kafka.log.message => message
Document kafka module file migrations in ecs-migration
e8dc178
Alias two Kafka fields. Remove unused field definition.
7a4383b 
The pipeline ends with remove `remove field kafka.log.timestamp`.
Changelog
64316a1
@webmat webmat merged commit c053a32 into elastic:master Dec 20, 2018
@webmat webmat deleted the ecs-kafka-fb branch December 20, 2018 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin approved these changes

Assignees

@webmat webmat

Labels

ecs Filebeat Filebeat module review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@webmat @ruflin