Field type conflict blocks the output buffer #2245
Description
High level description: A type mismatch for a single point totally blocks the output buffer for outputs.influxdb.
Version tested: v1.1.2
input plugins: any
output plugins: outputs.influxdb in HTTP mode
Conditions:
A field key for a measurement already exists with a defined context type (e.g. float) in the backend InfluxDB
A metric arrives to Telegraf for that measurement & field with a different context type (e.g. int)
Issue:
The output queue gets stuck due to type mismatch. Telegraf indefinitely retries to write the mismatching point and does not flush the output buffer at all. Any following points are stacked up and are not written to InfluxDB.
e.g., using the Telegraf inputs.http_listener:
curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1"
curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1i"
2017/01/10 13:46:46 I! Output [influxdb] buffer fullness: 1 / 10000 metrics. Total gathered metrics: 2. Total dropped metrics: 0.
2017/01/10 13:46:46 E! InfluxDB Output Error: {"error":"field type conflict: input field \"value\" on measurement \"test\" is type integer, already exists as type float"}
curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1"
2017/01/10 13:47:08 I! Output [influxdb] buffer fullness: 2 / 10000 metrics. Total gathered metrics: 3. Total dropped metrics: 0.
2017/01/10 13:47:08 E! InfluxDB Output Error: {"error":"field type conflict: input field \"value\" on measurement \"test\" is type integer, already exists as type float"}
Result: Output buffer fullness increases indefinitely.
Sample config used for the above test:
[[inputs.http_listener]]
service_address = ":8186"
read_timeout = "10s"
write_timeout = "10s"
max_body_size = 0
max_line_size = 0
[[outputs.influxdb]]
urls = ["http://localhost:8086"] # required
database = "telegraf" # required
retention_policy = ""
write_consistency = "any"
timeout = "5s"
Proposal: add a max-attempts parameter for type conflicts to avoid indefinitely blocking the buffer.