Skip to content

[WIP] Filebeat module issues found during ECS conversion #9208

@webmat

Description

@webmat

Please ignore for now.

This is just a collection of all the small problems we notice could be improved while performing the ECS transition (#8655). Some things are small enough that we can tackle them right in the ECS PRs.

Some things are out of scope, though. Here they are:

Usage of http.response.elapsed_time

This field is not in ECS and should be deprecated in favour of event.duration. Modules using it:

  • kibana

Full version strings vs breakdown fields

  • Ingest Node's UA parser gives us the version number in broken up in 3 fields (major/minor/patch), but no full version string (e.g. 10.14.1). We should populate user_agent.version and user_agent.os.version with full version strings. Modules affected:
    • apache2.access
    • iis.access
    • nginx.access
    • traefik.access

system.syslog

  • Add integration test for the convert_timezone option

system.auth

  • Add integration tests for message formats other than sshd, sudo, groupadd and useradd. Right now it's only testing for those, and no test for the last GROK pattern that just populates the message field.
  • Patterns specific to sshd, sudo, groupadd and useradd
  • After performing field renames on auth messages, I wonder if the resulting events are missing something. For example the first pattern uses only ECS fields, if it wasn't for one last field that hasn't been renamed (system.auth.ssh.method, line 29), I would never know that its ECS event is a message about SSH.

nginx.access

  • Current parsing of access log assumes people are using $http_x_forwarded_for rather than the default $remote_addr for their remote address. This means even though nginx can receive connections from unix sockets (e.g. common when polling for nginx stats), this module doesn't support these log events.

Message in error filesets

E.g. apache.error, nginx.error

  • Currently outputs to message rather than error.message, since all Ingest Node documentation currently suggests outputting Ingest Node error messages at error.message.
    • I'm not 100% convinced about using error.message in any case for userland error messages, by the way. We've been talking about always populating message for the timeline viewer.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions