Skip to content

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

@andrewkroh

Description

@andrewkroh

The Elastic support matrix indicates that the latest Filebeat 7.x version works with all 7.x versions of Elasticsearch. This is an assumption I'm making based on the table pictured below. There is a "Compatibility with Beats" table but it does not include Elasticsearch or Kibana columns.

Screen Shot 2021-08-10 at 8 50 28 AM

This is not true for all Filebeat modules because there are specific Elasticsearch features utilized in module ingest node pipelines that require newer ES versions. For example:

  • The set processors copy_from feature. Used to copy objects.
  • The convert processor's type: ip. Used to ensure strings are valid IPs for ip mapping fields.
  • registered_domain is new in 7.13.
  • network_direction is new in 7.12.

There are several other features where Filebeat automatically rewrites the ingest node pipeline to create a compatible pipeline for the current ES version (possibly with a reduced feature set).

  • The append processors allow_duplicates.
  • community_id is new in 7.12
  • The set processor's ignore_empty.
  • uri_parts is new in 7.12
  • user_agent requires ecs: true in ES [6.7, 7.0).

How do we want to handle modules that require specific Elasticsearch versions and how do we indicate this to users in the support matrix?

  • It might just be to say a particular module requires ES 7.N, documented it, add a runtime check, and then put a footnote in the support matrix that says certain Filebeat modules have additional version constraints.
  • Or indicate that Filebeat requires a specific ES version like Filebeat 7.13 requires at least ES 7.12 (FB works with previous ES minor). This would mean that modules can be developed using more recent ES features and it's easy to test with one previous ES version.
  • Manually fix every module pipeline that uses new features like copy_from or convert with type: ip. With this we never get to take advantage of new features in ES.
  • Try to automatically rewrite pipelines for compatibility. This solution is hard to maintain and test for every edge case. And some configs cannot be automatically rewritten. I attempted this in [Filebeat] Make set processor with copy_from compatible with ES < 7.13 #26593.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions