New JSON Parser by sspaink · Pull Request #9246 · influxdata/telegraf

sspaink · 2021-05-06T18:23:05Z

Updated associated README.md.
Wrote appropriate unit tests.

This pull request introduces a new JSON Parser called json_v2 that is intended to be more flexible then the existing JSON parser.

The query language being used will still be GJSON Path Syntax, but the way to configure this parser has changed significantly. I've provided multiple configuration examples within the testdata directory under the json_v2 directory. The README.md also goes into more detail on how the configuration works.

Issues closed:

Allow setting tags inside or outside of a nested array
- Closes: JSON Nested Key Tags #6437 (The testdata directory named "nested_and_nonnested_tags" uses the JSON from this issue)
- Closes: plugins/parsers/json: tag_keys can not be applied to sub-objects #6853 (The testdata directory named "nested_tags" and "nested_tags_complex" use the JSON from this issue)
Fields in the JSON can be of any type (int,float,string,bool)
- Closes: JSON Parser for Boolean requires "json_string_fields" #7097 (The testdata directory named "types" covers this issue)
Set measurement name independently from other queries and allow any type
- Closes: Cannot use both json_name_key and json_query in mqtt_consumer plugin #6990 (The testdata directory named "nested_array_of_objects" uses the JSON from this issue)
- Closes: Json Parser User Number as json_name_key #7212 (The testdata directory named "measurement_name_int" covers this issue
Set timestamp independently from other queries
- Closes: Propsal regarding telegraf json_time_key/json_query configuration usage #5940 (the testdata directory named "timestamp" covers this issue)

telegraf-tiger · 2021-05-16T23:00:42Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-05-18T00:14:02Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-05-18T20:21:09Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-02T03:04:33Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-02T21:11:58Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-03T17:33:26Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-03T17:52:25Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-03T18:42:54Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

samhld · 2021-06-03T19:06:01Z

@sspaink @sjwang90

This is so cool! I tested it out and it works great! I'll file the "zip" feature request today.

Two notes from testing:

I'm wondering if the syntax of [[xxx.xxx.json_v2.object]] and [[xxx.xxx.json_v2.tag]], etc. is unnecessarily verbose? In the examples, the inputs.file portion seems to be implied, right? Is there a technical reason we wouldn't revert back to simply [[object]], [[tag]], etc.? The TOML indentations seem to make it clear what the prefixes would be IMO.
If we do stick with the longer form, it might be good to explain what the xxx notation is in the example configurations (maybe just a comment // where 'xxx' is ...)

telegraf-tiger · 2021-06-03T19:47:11Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

sspaink · 2021-06-03T20:34:14Z

@samhld thanks for the feedback! This is a limitation of TOML that sub-tables needs the parent name. I agree it would be nicer if it was just object/field/tag. I did update the README to be more clear and removed "xxx" with an actual example.
Spec for reference: https://github.com/toml-lang/toml/blob/master/toml.md#table

samhld · 2021-06-03T20:41:00Z

@sspaink Oh gotcha! TIL!

telegraf-tiger · 2021-06-03T21:33:58Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-04T02:00:51Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-04T17:13:48Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-07T17:17:53Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-09T22:42:06Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-06-10T16:37:20Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

plugins/parsers/json_v2/README.md

reimda · 2021-06-10T18:12:59Z

plugins/parsers/json_v2/README.md

+* `uint`, bool, floats or strings (with valid numbers) can be converted to a uint.
+* `string`, any data can be formatted as a string.
+* `float`, string values (with valid numbers) or integers can be converted to a float.
+* `bool`, the string values "true" or "false" (regardless of capitalization) or the integer values `0` or `1`  can be turned to a bool.


Would it be worth it to add a section for people upgrading from v1?

Good idea, we should look into adding this. Need to get some example configs from v1.

reimda

Good job!

telegraf-tiger · 2021-06-10T19:18:38Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

(cherry picked from commit 885252d)

telegraf-tiger bot added new plugin plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins labels May 6, 2021

sspaink force-pushed the jsonparser branch from 8b8a534 to 12999f1 Compare May 16, 2021 05:33

influxdata deleted a comment from telegraf-tiger bot May 16, 2021

sspaink changed the title ~~New JSON Parser using JSONpath~~ New JSON Parser May 17, 2021

srebhan mentioned this pull request May 19, 2021

XPath parser extension to allow parsing of JSON, MessagePack and Protocol-buffers #9277

Merged

2 tasks

sspaink force-pushed the jsonparser branch from f55c637 to df589d5 Compare June 3, 2021 17:38

sspaink marked this pull request as ready for review June 4, 2021 01:47

sspaink force-pushed the jsonparser branch from cbb799d to 9ded300 Compare June 4, 2021 17:26

Sebastian Spaink added 2 commits June 10, 2021 11:17

First commit

2264ae3

Test progress

9fa9ab5

Sebastian Spaink added 19 commits June 10, 2021 11:17

Add example with multiple arrays

99319a6

Fix ordering logic for appending

1f88ac8

Cleanup comments and var names

7658bdb

Set measurement name with query

2468401

Add test for measurement name as int

f2da9ab

Rename fields and add prepend key support

f57debb

Cleanup readme

c45cf96

cleanup example config

d61da46

Add missing key in README

969b6a6

typo

0b708ae

Fix bug setting tag and type

da2f0f8

Resolve merge conflict

57718b8

Update README

d50b440

Update README

1539e15

Update included keys to allow nested keys

b0c015f

refactor

7fccad8

Update README

0f23e6a

go mod tidy

3cbfd19

Support multiple timestamps in object

553f8c9

sspaink force-pushed the jsonparser branch from be29795 to 553f8c9 Compare June 10, 2021 16:20

go mod tidy

f6eb147

reimda reviewed Jun 10, 2021

View reviewed changes

http to file

26f5fb4

reimda approved these changes Jun 10, 2021

View reviewed changes

reimda merged commit 885252d into influxdata:master Jun 10, 2021

reimda linked an issue Jun 10, 2021 that may be closed by this pull request

Support JSON Path-like json parsing #1363

Closed

reimda pushed a commit that referenced this pull request Jun 10, 2021

New JSON Parser (#9246)

0e63897

(cherry picked from commit 885252d)

arstercz pushed a commit to arstercz/telegraf that referenced this pull request Aug 28, 2025

New JSON Parser (influxdata#9246)

6b0832c

Conversation

sspaink commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

telegraf-tiger bot commented May 16, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented May 18, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented May 18, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 2, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 2, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 3, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 3, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 3, 2021

Artifact URLs

Uh oh!

samhld commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

telegraf-tiger bot commented Jun 3, 2021

Artifact URLs

Uh oh!

sspaink commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samhld commented Jun 3, 2021

Uh oh!

telegraf-tiger bot commented Jun 3, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 4, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 4, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 7, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 9, 2021

Artifact URLs

Uh oh!

telegraf-tiger bot commented Jun 10, 2021

Artifact URLs

Uh oh!

Uh oh!

reimda Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

sspaink Jun 10, 2021

Choose a reason for hiding this comment

Uh oh!

reimda left a comment

Choose a reason for hiding this comment

Uh oh!

telegraf-tiger bot commented Jun 10, 2021

Artifact URLs

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sspaink commented May 6, 2021 •

edited

Loading

samhld commented Jun 3, 2021 •

edited

Loading

sspaink commented Jun 3, 2021 •

edited

Loading