Skip to content

Improve user experience when using a custom ES index #3691

@monicasarbu

Description

@monicasarbu

In case the user wants to use a custom index in Elasticsearch, we should provide a way for the user to configure the index name and have the Beat take care of adjusting the template, dashboards and the index where to insert the data.

What's available
Right now, if the user wants to use a custom index the following steps are done:

  • edit the template to use the right ES index name
  • during setup (-setup command line option), the user needs to specify the -E "dashboards.index=new_index" to adjust the Kibana index pattern and dashboards.

The issues

  1. As of Dynamically generate template on startup #3681, the Beat will generate the template at startup, and there is not possible to edit the template anymore. So, we need to generate the template to include the right configured index pattern.

  2. Where do we configure the ES index name? Right now, you can configure the custom index when loading the dashboards under dashboards.index, and it will change the index pattern and the dashboards. You can also configure the index where the data is stored in Elasticsearch:
    underoutput.elasticsearch.index in case ES output is enabled, under output.logstash.index in case of Logstash output.

In conclusion, there are different configuration options to configure the same index name. It would be nice to unify them into a single configuration option, maybe outside the output section.

cc-ed @ruflin @tsg

Metadata

Metadata

Assignees

No one assigned

    Labels

    StalleddiscussIssue needs further discussion.enhancementneeds_teamIndicates that the issue/PR needs a Team:* label

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions