Convert Filebeat iis.access to ECS by webmat · Pull Request #9084 · elastic/beats

webmat · 2018-11-14T21:33:15Z

Caveats

user agent is encoded in the log. Decoding to get better UA parsing results.
- It's not quoted, so spaces are replaced with + signs. Parens and slashes are not replaced, though.
- Using urldecode on it makes it more palatable to user_agent parser.

Renames

iis.access.server_ip => destination.ip
iis.access.remote_ip => source.ip
iis.access.method => http.request.method
iis.access.url => url.path
iis.access.query_string => url.query
iis.access.port => destination.port
iis.access.user_name => user.name
iis.access.referrer => http.request.referrer
iis.access.response_code => http.response.status_code
iis.access.hostname => destination.domain
iis.access.user_agent.original => user_agent.original
iis.access.geoip => source.geo

TODO

Convert status code, port, byte counts, request times to int
Update ECS-migration.yml file
Changelog
Create field aliases
Output user_agent to ECS field names
- Next step: figure out what to do with fully broken down OS version numbers, when ECS supports one version string :-)
List decoding of user agent in breaking changes

ruflin

Add changelog and migration file.

filebeat/module/iis/access/test/test.log-expected.json

webmat · 2018-11-20T13:46:43Z

@ruflin Ready for review.

ruflin

For user_agent problem, see other issue.

ruflin · 2018-11-20T13:50:24Z

filebeat/module/iis/access/ingest/default.json

IPORHOST here seems also one of these fields where we should do a follow up PR to make it only IP as otherwise it could break on ingest time.

filebeat/module/iis/access/ingest/default.json

webmat · 2018-11-21T22:16:32Z

@ruflin I've worked a bit on the user_agent parsing. You were right about the mapping issue. So this is resolved. My work has revealed a few things:

I noticed that in IIS logs, the UA is not quoted, and they therefore replace spaces with the +. I'm adding a urldecode step to get more precise UA parsing. (introduced in 6.1)
I added a different UA in the test log, which does get it's OS major and minor version parsed successfully. This revealed a problem, outlined at next point.
UA parser gives us fully broken down version fields (major, minor, patch), but no whole "version" field, whereas in ECS we only define "version".
- This is showing up as a problem for user_agent.os.version. For now I haven't fixed this issue yet, I only created fields temporarily, so the tests don't break for this.

I think there's many ways we can go about this.

ECS should likely add support for version breakdown fields
Tackling the user_agent in this PR is informative. But we may want to close the other PRs without the UA fix, and do a subsequent PR only about getting UA right in all access logs.

- iis.access.server_ip => destination.ip - iis.access.remote_ip => source.ip - iis.access.method => http.request.method - iis.access.url => url.path - iis.access.query_string => url.query - iis.access.port => destination.port - iis.access.user_name => user.name - iis.access.referrer => http.request.referrer - iis.access.response_code => http.response.status_code - iis.access.hostname => destination.domain - iis.access.user_agent.original => user_agent.original - iis.access.geoip => source.geo

…veals a bug

…his works.

webmat · 2018-11-22T19:09:11Z

This is ready for a final review.

Please make sure to check out the caveat above. I've made sure to list this "breaking" change improvement separately from the ECS transition.

ruflin · 2018-11-22T20:33:31Z

For the version: I prefer the full version string. We had it broken down in an early version of ECS but I didn't see the benefits. Elasticsearch offers prefix queries which means queries for major.minor as an example are possible. The only thing not possible is for example > major but I would rather see ES have a type version which would support all these type of queries.

ruflin

LGTM. Ready for merge.

For the user_agent decoding: Do we need this only for IIS or also in other modules?

ruflin · 2018-11-22T20:37:33Z

filebeat/module/iis/access/_meta/fields.yml

-          description: >
-            The major version of the operating system.
+          type: alias
+          path: user_agent.os.major


We could probably use script processor to concatenate the version fields. Initially hoped the the join processor would do this but seems it has a different purpose.

Yeah, but all of this refining needs to go in a shared pipeline. We're not going to duplicate this work everywhere, on the eve of starting to share pipelines ;-)

webmat · 2018-11-22T21:22:05Z

For the user_agent decoding: Do we need this only for IIS or also in other modules?

No, in all of the open source web servers modules I've been working on, the user agent was always a quoted string. None of them needed to encode all the spaces by replacing them with a +. IIS is the only one where I've seen that.

webmat · 2018-11-22T21:36:18Z

Sorry, had missed this comment #9084 (comment) before reading your review.

Reconstructing the version string is out of scope for this PR, IMO. The user_agent IN processor doesn't give it to us, at this time. I'm down with requesting that change to the IN team, or reconstructing it, but I don't think it should block this mapping to ECS.

I've added this to #9208 so we don't lose track of this.

…ccess logs (#9955) - Introduce IPv6 zone workaround iis.access log as well, resolving #9836. - Update the IPv6 zone fix (#9869) for iis.error to use the ECS `.address` field instead of a transient field. - Convert many fields under `iis.error.*` to ECS. Previous field names are field aliases towards the new corresponding ECS field: - iis.error.remote_ip => source.address - iis.error.remote_port => source.port - iis.error.server_ip => destination.address - iis.error.server_port => destination.port - iis.error.http_version => http.version - iis.error.method => http.request.method - iis.error.url => url.original - iis.error.response_code => http.response.status_code - iis.error.geoip.* => source.geo.* - read_timestamp => event.created (not aliased, still used elsewhere) - Update field aliases introduced in #9084 to point to `.address` instead of `.ip`, since this value can be ambiguous. The IP field is populated with the cleaned up IP without the zone. This is also true for the `.ip` fields under populated by the error logs. - iis.access.remote_ip => source.address - iis.access.server_ip => destination.address - Coerce to long: source.port, destination.port and http.response.status_code in the iis.error fileset

ruflin mentioned this pull request Nov 14, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 14, 2018

webmat requested a review from ruflin November 14, 2018 21:33

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 14, 2018

ruflin reviewed Nov 19, 2018

View reviewed changes

filebeat/module/iis/access/test/test.log-expected.json Outdated Show resolved Hide resolved

webmat force-pushed the ecs-iis-access branch from 5afd5c5 to 1d16d14 Compare November 20, 2018 03:53

webmat removed the in progress Pull request is currently in progress. label Nov 20, 2018

ruflin added the review label Nov 20, 2018

ruflin changed the title ~~[WIP] Convert Filebeat iis.access to ECS~~ Convert Filebeat iis.access to ECS Nov 20, 2018

ruflin reviewed Nov 21, 2018

View reviewed changes

webmat force-pushed the ecs-iis-access branch from 2de8a2a to a8e1881 Compare November 21, 2018 22:08

Mathieu Martin added 12 commits November 22, 2018 14:00

Convert ints to ints

a78d499

Shove all renames so far in ecs-migration.yml

7e4ab6e

Changelog

befbc8b

Add http.request.referrer to this PR as well

edb2ae0

Output successfully to user_agent

d203691

URL decode the user agent string, to get more detailed UA analysis

ac46086

Introduce user agent whose version gets parsed successfully, which re…

eac7bb6

…veals a bug

Add temporary field defs to go past this failure for now

8bd9fcf

Rename so the version breakdown looks like ua.os.major/minor/patch. T…

108a02a

…his works.

Replace previous definitions with aliases for all normalized fields

db4191c

Move non-normalized field defs at the top, to promote sanity.

c36be2e

webmat force-pushed the ecs-iis-access branch from a8e1881 to c36be2e Compare November 22, 2018 19:08

Add breaking change about decoding the user agent.

2f98beb

ruflin approved these changes Nov 22, 2018

View reviewed changes

webmat merged commit 04c951d into elastic:master Nov 22, 2018

webmat deleted the ecs-iis-access branch November 22, 2018 21:44

webmat mentioned this pull request Dec 6, 2018

Re-introduce a simplified version of user_agent. elastic/ecs#240

Merged

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

Conversation

webmat commented Nov 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

webmat commented Nov 20, 2018

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

ruflin Nov 20, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

webmat commented Nov 21, 2018

Uh oh!

webmat commented Nov 22, 2018

Uh oh!

ruflin commented Nov 22, 2018

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

ruflin Nov 22, 2018

Choose a reason for hiding this comment

Uh oh!

webmat Nov 22, 2018

Choose a reason for hiding this comment

Uh oh!

webmat commented Nov 22, 2018

Uh oh!

webmat commented Nov 22, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

webmat commented Nov 14, 2018 •

edited

Loading