Skip to content

Finish migration to event.duration for 4 Filebeat modules#10274

Merged
webmat merged 5 commits intoelastic:masterfrom
webmat:ecs-finish-event-duration
Jan 24, 2019
Merged

Finish migration to event.duration for 4 Filebeat modules#10274
webmat merged 5 commits intoelastic:masterfrom
webmat:ecs-finish-event-duration

Conversation

@webmat
Copy link
Copy Markdown
Contributor

@webmat webmat commented Jan 23, 2019 •
edited
Loading

This PR is built on top of #10188 and should not be merged before it.

Please only review commits from Jan 23rd or later.

Filebeat modules affected

  • elasticsearch.slowlog
  • haproxy.log
  • mysql.slowlog
  • postgresql.log

Introduced in this PR

  • All pipelines use a temp value and a common Ingest Node painless processor, to reduce script compilations.
    • MySQL uses temp field but not shared code, as it requires a tweak to avoid float imprecision problems
  • Old field names removed from fields definitions. Since the scale changes by a factor or 1 million or 1 billion, we can't make them aliases with a straight face.
  • Each of these migrated fields gets a special entry in ecs-migration.yml, which mentions the scale difference between old field and event.duration, and marks them as alias: false.

@webmat webmat requested review from a team as code owners January 23, 2019 05:35
@webmat webmat self-assigned this Jan 23, 2019
@webmat webmat requested a review from ruflin January 23, 2019 05:54
@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 23, 2019

@ycombinator Tweak to event.duration handling for elasticsearch.slowlog

ycombinator
ycombinator approved these changes Jan 23, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ES slowlog fileset change LGTM. Since it's a breaking change – the elasticsearch.slowlog.took_millis field is no longer being indexed — worth documenting in the breaking changes section of the CHANGELOG?

ruflin
ruflin approved these changes Jan 23, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruflin ruflin mentioned this pull request Jan 23, 2019
Closed
@webmat webmat force-pushed the ecs-finish-event-duration branch from abbebd8 to d975b1f Compare January 23, 2019 18:16
Changelog
3a15bf2 
Trying to keep chattiness to a minimum by listing 2 PRs on that breaking change line.
@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 23, 2019

Adding to a changelog like this: kosher?

@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 23, 2019

Failing libbeat test seems unrelated to me. No field changes here could cause a problem, AFAIK.

@webmat webmat merged commit 11f9116 into elastic:master Jan 24, 2019
@webmat webmat deleted the ecs-finish-event-duration branch January 24, 2019 03:08
@sakurai-youhei sakurai-youhei mentioned this pull request Jun 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator ycombinator approved these changes

@ruflin ruflin ruflin approved these changes

Assignees

@webmat webmat

Labels

ecs Filebeat Filebeat module review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@webmat @ycombinator @ruflin