Skip to content

Update HTTP protocol to use ECS fields#9976

Merged
andrewkroh merged 3 commits intoelastic:masterfrom
andrewkroh:feature/pb/http-ecs
Jan 16, 2019
Merged

Update HTTP protocol to use ECS fields#9976
andrewkroh merged 3 commits intoelastic:masterfrom
andrewkroh:feature/pb/http-ecs

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

NOTE: This is based on #9941 so ignore the first two commits. They will be removed after that PR merges.

Here's a summary of what fields changed.

Changed

  • bytes_in -> source.bytes
  • bytes_out -> destination.bytes
  • http.request.body -> http.request.body.content
  • http.response.body -> http.response.body.content
  • http.response.code -> http.response.status_code
  • http.response.phrase -> http.response.status_phrase
  • method -> http.request.method (lowercased)
  • notes -> error.message
  • params -> url.query
  • path -> url.path
  • real_ip -> network.forwarded_ip
  • responsetime -> event.duration (unit are now nanoseconds)
  • transport -> network.transport

Added

  • event.dataset = http
  • event.end
  • event.start
  • http.request.referrer (always added if Referer header is present)
  • http.version
  • network.bytes
  • network.community_id
  • network.protocol = http
  • network.type
  • source.domain (added if Host header is present and not an IP address)
  • url.domain - set with the Host header value
  • url.full (synthesized from data in the request/response)
  • url.port (when port is != 80)
  • user_agent.original - (always added if User-Agent header is present)

Unchanged Packetbeat Fields

  • query = {{ http.request.method }} {{ url.path }}
  • request - text representation of the entire request
  • response - text representation of the entire response
  • status
  • type = http (we might remove this since we have event.dataset)

@andrewkroh andrewkroh requested review from a team as code owners January 9, 2019 23:16
@andrewkroh andrewkroh force-pushed the feature/pb/http-ecs branch 3 times, most recently from 0873437 to a085e44 Compare January 10, 2019 06:00
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this become an alias type with migration: true? Same for the other fields which were removed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this is what the ecs-migration's alias: true was going to be used for. So for any aliases that I want I should put them in fields.yml?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you need to put them in both places.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added aliases for the HTTP fields where possible.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be event.duration?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should. Despite the huge diff I only changed the "Web Transactions" visualization of the overview dashboard. I will cycle back towards the end of this effort and do all of the general parts of the overview dashboard.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would hope this PR does it automatically for you: #9998

@andrewkroh andrewkroh force-pushed the feature/pb/http-ecs branch 3 times, most recently from 4212936 to d6f1045 Compare January 15, 2019 14:49
Here's a summary of what fields changed.

Part of elastic#7968

Changed

- bytes_in -> source.bytes
- bytes_out -> destination.bytes
- http.request.body -> http.request.body.content
- http.response.body -> http.response.body.content
- http.response.code -> http.response.status_code
- http.response.phrase -> http.response.status_phrase
- method -> http.request.method (lowercased)
- notes -> error.message
- params -> url.query
- path -> url.path
- real_ip -> network.forwarded_ip
- responsetime -> event.duration (unit are now nanoseconds)
- transport -> network.transport

Added

- event.dataset = http
- event.end
- event.start
- http.request.referrer (always added if Referer header is present)
- http.version
- network.bytes
- network.community_id
- network.protocol = http
- network.type
- source.domain (added if Host header is present and not an IP address)
- url.domain - set with the Host header value
- url.full (synthesized from data in the request/response)
- url.port (when port is != 80)
- user_agent.original - (always added if User-Agent header is present)

Unchanged Packetbeat Fields

- query = {{ http.request.method }} {{ url.path }}
- request - text representation of the entire request
- response - text representation of the entire response
- status
- type = http (we might remove this since we have event.dataset)

The HTTP dashboard was updated too.
@andrewkroh
Copy link
Copy Markdown
Member Author

Rebased to resolve an ecs-migration.yml conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants