Skip to content

outputs.influxdb writing to missing databases or retentions result in endless retries #9211

@passing

Description

@passing

Relevant telegraf.conf:

[agent]
    flush_interval = "1s"

[[inputs.influxdb_listener]]
    database_tag = "__database__"
    retention_policy_tag = "__retention__"

[[outputs.influxdb]]
    urls = ["http://influxdb:8086"]
    skip_database_creation = true
    database_tag = "__database__"
    exclude_database_tag = true
    retention_policy_tag = "__retention__"
    exclude_retention_policy_tag = true

[[outputs.file]]
    files = ["stdout"]

Reproduce the issue

with the telegraf.conf file, execute the following to run an influxdb and telegraf in docker:

docker run -d --rm --name influxdb -p 8086:8086 influxdb:1.8.3
curl "http://localhost:8086/query" -d "q=CREATE DATABASE telegraf WITH DURATION 1d"

docker run --rm -ti --link influxdb -v $(pwd):/etc/telegraf -p 8186:8186 telegraf:1.18.1

# to cleanup later, run: docker stop influxdb

now metrics can be sent like this

curl "http://localhost:8186/write?db=telegraf" -d "test value=1"
curl "http://localhost:8186/write?db=telegraf&rp=autogen" -d "test value=1"

Problem 1

writing to an invalid database:

curl "http://localhost:8186/write?db=invalid" -d "test value=1"

results in this error repeating forever

2021-04-28T12:37:11Z E! [outputs.influxdb] When writing to [http://influxdb:8086]: 404 Not Found: database not found: "invalid"
2021-04-28T12:37:11Z E! [agent] Error writing to outputs.influxdb: could not write any address

Problem 2

writing to an invalid retention:

curl "http://localhost:8186/write?db=telegraf&rp=invalid" -d "test value=1"

results in this error repeating forever

2021-04-28T12:37:32Z E! [outputs.influxdb] When writing to [http://influxdb:8086]: 500 Internal Server Error: retention policy not found: invalid
2021-04-28T12:37:32Z E! [agent] Error writing to outputs.influxdb: could not write any address

Expected behavior:

it would be expected, that when these errors occur, the metrics get dropped and don't get retried forever (while the telegraf buffer is filling up more and more)

Actual behavior:

metrics fill up the buffer of telegraf and writing to influx is retried forever

Metadata

Metadata

Assignees

Labels

area/influxdbbugunexpected problem or unintended behaviorgood first issueThis is a smaller issue suited for getting started in Telegraf, Golang, and contributing to OSS.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions