Skip to content

elasticsearch output: test to show a bug handling NaN#9221

Closed
adrianlzt wants to merge 1 commit intoinfluxdata:masterfrom
adrianlzt:bug/elasticsearch_nan
Closed

elasticsearch output: test to show a bug handling NaN#9221
adrianlzt wants to merge 1 commit intoinfluxdata:masterfrom
adrianlzt:bug/elasticsearch_nan

Conversation

@adrianlzt
Copy link
Copy Markdown
Contributor

Elasticsearch output is not able to handle "NaN" values, which are valid
values (at least generated by inputs prometheus)

This new test shows how output breaks if it receives a NaN value.

NaN values are valid, at least contemplated in the prometheus tests.

I don't provide a fix because I am not sure how this wants to be handled.

Elasticsearch output is not able to handle "NaN" values, which are valid
values (at least generated by inputs prometheus)
@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label Apr 29, 2021
@adrianlzt
Copy link
Copy Markdown
Contributor Author

json serializer ignores math.NaN and Inf and values.

			// JSON does not support these special values
			if math.IsNaN(fv) || math.IsInf(fv, 0) {

@adrianlzt
Copy link
Copy Markdown
Contributor Author

Workaround, drop NaN values.

[[processors.starlark]]
  source = '''
load("logging.star", "log")

def apply(metric):
    for k, v in metric.fields.items():
        if type(v) == "float" and str(v) == "NaN":
            metric.fields.pop(k)
            log.warn("Dropped NaN value: metric {} field {}".format(metric.name, k))

    return metric
'''

@adrianlzt
Copy link
Copy Markdown
Contributor Author

Test are passing because test avoided in the short mode.

The result of the test:

➜ go test -v -run TestConnectAndWriteMetricWithNaNValue
=== RUN   TestConnectAndWriteMetricWithNaNValue
2021/04/29 20:11:09 I! Elasticsearch version: 7.1.1
2021/04/29 20:11:09 D! Elasticsearch template telegraf created or updated
    elasticsearch_test.go:67: 
                Error Trace:    elasticsearch_test.go:67
                Error:          Received unexpected error:
                                Error sending bulk request to Elasticsearch: json: unsupported value: NaN
                Test:           TestConnectAndWriteMetricWithNaNValue
--- FAIL: TestConnectAndWriteMetricWithNaNValue (0.13s)
FAIL
exit status 1
FAIL    github.com/influxdata/telegraf/plugins/outputs/elasticsearch    0.133s

@srebhan
Copy link
Copy Markdown
Member

srebhan commented Oct 18, 2021

@adrianlzt interesting. Sorry for having this sitting around for sooo long. Do you also have a fix for the issue? For example, checking explicitly for NaN or inf in the serialization?

@srebhan srebhan self-assigned this Oct 18, 2021
@adrianlzt
Copy link
Copy Markdown
Contributor Author

I don't have a fix, sorry

@srebhan
Copy link
Copy Markdown
Member

srebhan commented Nov 30, 2021

@adrianlzt can you please check if #10196 fixes your issue!?

@srebhan
Copy link
Copy Markdown
Member

srebhan commented Dec 8, 2021

Ok bugfix is in. Closing this PR... Thanks for your contribution @adrianlzt!

@srebhan srebhan closed this Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/elasticsearch fix pr to fix corresponding bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants