Relevant telegraf.conf:
[agent]
flush_interval = "1s"
[[inputs.influxdb_listener]]
database_tag = "__database__"
retention_policy_tag = "__retention__"
[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
skip_database_creation = true
database_tag = "__database__"
exclude_database_tag = true
retention_policy_tag = "__retention__"
exclude_retention_policy_tag = true
[[outputs.file]]
files = ["stdout"]
Reproduce the issue
with the telegraf.conf file, execute the following to run an influxdb and telegraf in docker:
docker run -d --rm --name influxdb -p 8086:8086 influxdb:1.8.3
curl "http://localhost:8086/query" -d "q=CREATE DATABASE telegraf WITH DURATION 1d"
docker run --rm -ti --link influxdb -v $(pwd):/etc/telegraf -p 8186:8186 telegraf:1.18.1
# to cleanup later, run: docker stop influxdb
now metrics can be sent like this
curl "http://localhost:8186/write?db=telegraf" -d "test value=1"
curl "http://localhost:8186/write?db=telegraf&rp=autogen" -d "test value=1"
Problem 1
writing to an invalid database:
curl "http://localhost:8186/write?db=invalid" -d "test value=1"
results in this error repeating forever
2021-04-28T12:37:11Z E! [outputs.influxdb] When writing to [http://influxdb:8086]: 404 Not Found: database not found: "invalid"
2021-04-28T12:37:11Z E! [agent] Error writing to outputs.influxdb: could not write any address
Problem 2
writing to an invalid retention:
curl "http://localhost:8186/write?db=telegraf&rp=invalid" -d "test value=1"
results in this error repeating forever
2021-04-28T12:37:32Z E! [outputs.influxdb] When writing to [http://influxdb:8086]: 500 Internal Server Error: retention policy not found: invalid
2021-04-28T12:37:32Z E! [agent] Error writing to outputs.influxdb: could not write any address
Expected behavior:
it would be expected, that when these errors occur, the metrics get dropped and don't get retried forever (while the telegraf buffer is filling up more and more)
Actual behavior:
metrics fill up the buffer of telegraf and writing to influx is retried forever
Relevant telegraf.conf:
Reproduce the issue
with the
telegraf.conffile, execute the following to run an influxdb and telegraf in docker:now metrics can be sent like this
Problem 1
writing to an invalid database:
results in this error repeating forever
Problem 2
writing to an invalid retention:
results in this error repeating forever
Expected behavior:
it would be expected, that when these errors occur, the metrics get dropped and don't get retried forever (while the telegraf buffer is filling up more and more)
Actual behavior:
metrics fill up the buffer of telegraf and writing to influx is retried forever