Skip to content

Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs#9955

Merged
webmat merged 10 commits intoelastic:masterfrom
webmat:ecs-iis-error-fb
Jan 11, 2019
Merged

Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs#9955
webmat merged 10 commits intoelastic:masterfrom
webmat:ecs-iis-error-fb

Conversation

@webmat
Copy link
Copy Markdown
Contributor

@webmat webmat commented Jan 8, 2019 •
edited
Loading

Caveats

  • Now filtering zone from both IP fields. Using and keeping the .address field to keep the raw address around, and .ip field is only set with the IP value without the zone.
  • Realized that the access logs were still vulnerable to the Microsoft IPv6 zone literal problem. Fixed it in this PR.

Renames

  • read_timestamp => event.created
  • iis.error.remote_ip => source.address
  • iis.error.remote_port => source.port
  • iis.error.server_ip => destination.address
  • iis.error.server_port => destination.port
  • iis.error.http_version => http.version
  • iis.error.method => http.request.method
  • iis.error.url => url.original
  • iis.error.response_code => http.response.status_code
  • iis.error.geoip.* => source.geo.*

Alias adjustments in iis.access

  • iis.access.remote_ip => source.address (instead of .ip)
  • iis.access.server_ip => destination.address (instead of .ip)

TODO

  • Add IPv6 zone support to access logs as well (like Handle IPv6 zone id in IIS filebeat ingest pipeline #9869 did for error logs)
  • Coerce some int fields
  • read_timestamp => event.created
  • Alias renamed fields to their ECS counterpart
  • Document field migrations in ecs-migration.yml
  • Revisit iis.access to fix the potential zone problem there too, using the .address fields
    • Fix the aliases to .address
  • Changelog

@webmat webmat requested a review from a team as a code owner January 8, 2019 19:24
@webmat webmat self-assigned this Jan 8, 2019
@webmat webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Jan 8, 2019
@webmat webmat requested a review from a team as a code owner January 8, 2019 20:05
@webmat webmat changed the title WIP Convert Filebeat iis.error to ECS Convert Filebeat iis.error to ECS Jan 8, 2019
@webmat webmat added review and removed in progress Pull request is currently in progress. labels Jan 8, 2019
@ruflin ruflin mentioned this pull request Jan 8, 2019
Closed
@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 8, 2019

jenkins, test this

@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 9, 2019

@ruflin ready for a review. Only Jenkins failures are flaky tests unrelated to Filebeat

@webmat webmat requested a review from ruflin January 9, 2019 20:17
@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Jan 10, 2019

I filed an issue here for the flaky test: #9987

@webmat webmat changed the title Convert Filebeat iis.error to ECS Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs Jan 10, 2019
@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 10, 2019

Rebased to fix a conflict.

This reminded me that in migrating iis.error to ECS, I also updated iis.access again, to tackle two things:

  • Support for Microsoft IPv6 zones (like Handle IPv6 zone id in IIS filebeat ingest pipeline #9869 did for error logs)
  • Use the dual .address + .ip fields. I did not initially use .address, but it was a great fit for the IP with zone, prior to copying the cleaned up IP address to the .ip field.

I've also updated the PR title to reflect this. @ruflin Ok with killing two birds with one PR? ;-)

ruflin
ruflin approved these changes Jan 11, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I resolved the changelog conflict.

@jsoriano
Copy link
Copy Markdown
Member

jsoriano commented Jan 11, 2019

Thanks for adding the IPv6 support to the access fileset too, this part would need to be backported as has been done with #9869.

@webmat webmat added the needs_backport PR is waiting to be backported to other branches. label Jan 11, 2019
@webmat
Copy link
Copy Markdown
Contributor Author

webmat commented Jan 11, 2019 •
edited
Loading

@jsoriano Yes, noted. I'll do that later today.

The backport will cover the support for zone in the IP address, but will not introduce the .address field, to be consistent.

Shall I create a brand new PR just for this on 6.x, and then backport to 6.6 as well, for 6.6.1? Or should I hold off for 6.6? I'm unsure about the exact timing of the procedure :-)

@urso urso removed the request for review from a team January 11, 2019 15:47
@webmat webmat force-pushed the ecs-iis-error-fb branch 2 times, most recently from c2ac4cd to 9a1c851 Compare January 11, 2019 18:47
@webmat webmat merged commit cf7b152 into elastic:master Jan 11, 2019
@webmat webmat deleted the ecs-iis-error-fb branch January 11, 2019 19:26
This was referenced Jan 11, 2019
Merged
Merged
@webmat webmat removed the needs_backport PR is waiting to be backported to other branches. label Feb 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin approved these changes

Assignees

@webmat webmat

Labels

ecs Filebeat Filebeat module review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@webmat @ruflin @jsoriano