Feature Request

Proposal:

Add a setting to Kafka output to limit the amount of metadata collected from Kafka broker.

Current behavior:

By default, Sarama collects all metadata for all topics: see Config > Metadata > Full is enabled by default.
This settings triggers a lot of Describe Topic operations which are blocked by Kafka ACLs:

[2021-09-22 16:01:11,008] INFO Principal = User:telegraf is Denied Operation = Describe from host = 192.138.1.54 on resource = Topic:LITERAL:metric_one (kafka.authorizer.logger)
[2021-09-22 16:01:11,013] INFO Principal = User:telegraf is Denied Operation = Describe from host = 192.168.2.48 on resource = Topic:LITERAL:metric_two (kafka.authorizer.logger)

The Kafka user named telegraf is only allowed to Describe the topic metric_telegraf in which it can also send data.
This may consume a lot of memory in Telegraf when Kafka has a lot of topics (several hundreds in our case).
The current setting raises a lot of errors on Kafka side (kafka-authorizer logs).

Desired behavior:

Introduce a setting in the Kafka output plugin which would allow to control Sarama's Config>Metadata>Full flag and disable it to limit the amount of Kafka metadata fetched by the Kafka client in Telegraf memory.

Metricbeat already does it properly: https://www.elastic.co/guide/en/beats/metricbeat/7.15/kafka-output.html#_metadata

Use case:

• Telegraf Kafka output should respect Kafka ACLs
• The default setting may consume a lot of Telegraf memory