inputs.mqtt/mqtt_consumer: allow connection errors on start #10694
Description
Feature Request
Proposal:
Telegraf should not crash when a single input fails to connect to its source. Ideally it would continue to retry the connection for that input, or permanently fail but continue running so that other inputs and outputs continue to work normally. There seem to be several bug reports related to this, including #3167 and #10078.
Current behavior:
A single telegraf mqtt_consumer input that fails to connect to an mqtt broker causes the entire telegraf service to shut down.
Example log (after the final log entry the telegraf service exits and logging ceases):
2022-02-20T23:53:35Z I! Starting Telegraf 1.21.4
2022-02-20T23:53:35Z I! Using config file: /etc/telegraf/telegraf.conf
2022-02-21T12:53:35+13:00 I! Loaded inputs: mqtt_consumer (2x)
2022-02-21T12:53:35+13:00 I! Loaded aggregators:
2022-02-21T12:53:35+13:00 I! Loaded processors:
2022-02-21T12:53:35+13:00 I! Loaded outputs: influxdb_v2 (2x)
2022-02-21T12:53:35+13:00 I! Tags enabled: host=telegraf-c9fc696bc-xb4r8
2022-02-21T12:53:35+13:00 I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf-c9fc696bc-xb4r8", Flush Interval:10s
2022-02-21T12:53:35+13:00 D! [agent] Initializing plugins
2022-02-21T12:53:35+13:00 D! [agent] Connecting outputs
2022-02-21T12:53:35+13:00 D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-02-21T12:53:35+13:00 D! [agent] Successfully connected to outputs.influxdb_v2
2022-02-21T12:53:35+13:00 D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-02-21T12:53:35+13:00 D! [agent] Successfully connected to outputs.influxdb_v2
2022-02-21T12:53:35+13:00 D! [agent] Starting service inputs
2022-02-21T12:57:45+13:00 E! [telegraf] Error running agent: starting input inputs.mqtt_consumer: network Error : EOF
Desired behavior:
Each input would be responsible for its own data source connection and not affect other inputs/outputs when the connection fails.
Use case:
The software is not usable in production without this functionality.