We use nsq_consumer to move messages from nsq to Influx
This setup has been running for ages (at least three years) but broke after upgrading to Telegraf 1.19.1; downgrading to 1.18.3 fixes the issue.
Relevant telegraf.conf:
[agent]
interval = "10s"
debug = true
[[inputs.nsq_consumer]]
nsqlookupd = ["nsqdev:4161"]
topic = "cisco-telemetry-influx"
channel = "to_influxdb_stor03"
data_format = "influx"
max_in_flight = 5000
[[outputs.influxdb]]
urls = [ "http://127.0.0.1:8086" ]
database = "cisco-telemetry"
System info:
- Telegraf version: 1.19.1 (also tested on 1.19.0 and 1.20.0)
- OS: Ubuntu 16.04 (also tested on Ubuntu 18.04)
Steps to reproduce:
2021-07-14T05:31:54Z I! Starting Telegraf 1.19.1
2021-07-14T05:31:54Z I! Loaded inputs: nsq_consumer
2021-07-14T05:31:54Z I! Loaded aggregators:
2021-07-14T05:31:54Z I! Loaded processors:
2021-07-14T05:31:54Z I! Loaded outputs: influxdb
2021-07-14T05:31:54Z I! Tags enabled: host=stor03
2021-07-14T05:31:54Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"stor03", Flush Interval:10s
2021-07-14T05:31:54Z D! [agent] Initializing plugins
2021-07-14T05:31:54Z D! [agent] Connecting outputs
2021-07-14T05:31:54Z D! [agent] Attempting connection to [outputs.influxdb]
2021-07-14T05:31:54Z D! [agent] Successfully connected to outputs.influxdb
2021-07-14T05:31:54Z D! [agent] Starting service inputs
2021-07-14T05:31:54Z D! [inputs.nsq_consumer] INF 1 [cisco-telemetry-influx/to_influxdb_stor03] querying nsqlookupd http://nsqdev:4161/lookup?topic=cisco-telemetry-influx
2021-07-14T05:31:54Z D! [inputs.nsq_consumer] INF 1 [cisco-telemetry-influx/to_influxdb_stor03] (nsq-dev:4150) connecting to nsqd
2021-07-14T05:31:54Z D! [inputs.nsq_consumer] WRN 1 [cisco-telemetry-influx/to_influxdb_stor03] (nsq-dev:4150) max RDY count 2500 < consumer max in flight 5000, truncation possible
2021-07-14T05:31:54Z D! [inputs.nsq_consumer] INF 1 [cisco-telemetry-influx/to_influxdb_stor03] () connecting to nsqd
2021-07-14T05:31:54Z E! [telegraf] Error running agent: starting input inputs.nsq_consumer: dial tcp: missing address
In this scenario nsqdev is a different host returned by nsqlookupd.
Expected behavior:
Telegraf should connect to the nsqd servers returned by nsqlookupd.
Actual behavior:
Telegraf fails with the error message Error running agent: starting input inputs.nsq_consumer: dial tcp: missing address
Additional info:
Data returned by nsqlookupd (curl http://nsqdev:4161/lookup?topic=cisco-telemetry-influx)
{"channels":["to_influxdb_stor03"],"producers":[{"remote_address":"10.20.30.134:46928","hostname":"nsq-dev","broadcast_address":"nsq-dev","tcp_port":4150,"http_port":4151,"version":"1.0.0-compat"}]}
We use nsq_consumer to move messages from nsq to Influx
This setup has been running for ages (at least three years) but broke after upgrading to Telegraf 1.19.1; downgrading to 1.18.3 fixes the issue.
Relevant telegraf.conf:
System info:
Steps to reproduce:
In this scenario nsqdev is a different host returned by nsqlookupd.
Expected behavior:
Telegraf should connect to the nsqd servers returned by nsqlookupd.
Actual behavior:
Telegraf fails with the error message
Error running agent: starting input inputs.nsq_consumer: dial tcp: missing addressAdditional info:
Data returned by nsqlookupd (
curl http://nsqdev:4161/lookup?topic=cisco-telemetry-influx){"channels":["to_influxdb_stor03"],"producers":[{"remote_address":"10.20.30.134:46928","hostname":"nsq-dev","broadcast_address":"nsq-dev","tcp_port":4150,"http_port":4151,"version":"1.0.0-compat"}]}