I'm writing this issue to raise a warning that Metricbeat and Filebeat are getting close to the point where users will not be able to create Kibana index patterns due to the number of fields. The Elasticsearch index mapping grows with each new module. The Kibana index-pattern is derived from the index mappings. Once the total size of the index-pattern reaches 1 MiB then you can no longer create Kibana index patterns.

There's probably room from one or two more Filebeat modules before this limit is reached. Some users may have already reached this limit if they are using dynamic fields of their own. In 7.9 development Filebeat accidentally hit this issue due to a field mapping mistake (see #19965).

What's affected?

The ability to use beat setup --dashboards to create an index pattern and the ability to create or refresh an index pattern through the Kibana Index patterns UI will be affected.

What's causing the limit?

As the number of fields increases, the size of the HTTP request body to create the index pattern through the Kibana API grows. Kibana limits requests to 1 MiB (as controlled by server.maxPayloadBytes in settings). The request to Kibana will fail with HTTP 413 Payload Too Large.

What are the sizes today?

• Collected from beat version 8.0.0 (amd64), libbeat 8.0.0 [b1bd7b7 built 2020-08-04 14:42:21 +0000 UTC].

How can we address it?

In no particular order here are some options:

• Increase the default server.maxPayloadBytes value.
• Increase max payload size just for the route where it's needed in 7.x only.
• Allow fields to be added to Kibana index patterns through partial updates (use a series of requests that are each less than 1 MiB to construct the index pattern).
• Minimize the number of new fields. Stop adding new modules. Stop adding new unused fields from ECS to the mapping.
• Please comment with your ideas.