Skip to content

Propsal regarding telegraf json_time_key/json_query configuration usage #5940

@yolkhovyy

Description

@yolkhovyy

Feature Request

This feature request is based on the following discussion: https://community.influxdata.com/t/telegraph-configuring-json-parsing-question/9518

Proposal:

Consider the following measurement in json format:

{
    "time": 1555745371410,
    "measurements": [
        {
            "name": "temperature",
            "value": 23.4,
            "units": "℃"
        },
        {
            "name": "moisture",
            "value": 5,
            "units": "%"
        },
        {
            "name": "light",
            "value": 10118,
            "units": "lux"
        },
        {
            "name": "fertility",
            "value": 0,
            "units": "us/cm"
        }
    ]
}

It is possible to parse/extract the time field with the following configuration:

[[inputs.mqtt_consumer]]
...
  data_format = "json"
  json_time_key = "time"
  json_time_format = "unix_ms"
...

It is also possible to parse/extract the measurements with the following configuration:

[[inputs.mqtt_consumer]]
...
  data_format = "json"
  json_query = "measurements"

  tag_keys = [
    "name",
    "units"
  ]
...

The latter will yield the following (which is quite nice):

> select * from mqtt_consumer
name: mqtt_consumer
time                host name        topic                                                         units value
----                ---- ----        -----                                                         ----- -----
1555745371450794118 **** temperature iot-2/type/TeMoLiFe/id/c67c8dfffe65cb63/evt/PubState/fmt/json ℃    24.3
1555745371450808740 **** moisture    iot-2/type/TeMoLiFe/id/c67c8dfffe65cb63/evt/PubState/fmt/json %     6
1555745371450815925 **** light       iot-2/type/TeMoLiFe/id/c67c8dfffe65cb63/evt/PubState/fmt/json lux   10168
1555745371450822476 **** fertility   iot-2/type/TeMoLiFe/id/c67c8dfffe65cb63/evt/PubState/fmt/json us/cm 58
1555745371505989365 **** temperature iot-2/type/TeMoLiFe/id/c67c8dfffe65d020/evt/PubState/fmt/json ℃    23.4
1555745371506015142 **** moisture    iot-2/type/TeMoLiFe/id/c67c8dfffe65d020/evt/PubState/fmt/json %     5
1555745371506029556 **** light       iot-2/type/TeMoLiFe/id/c67c8dfffe65d020/evt/PubState/fmt/json lux   10118
1555745371506043834 **** fertility   iot-2/type/TeMoLiFe/id/c67c8dfffe65d020/evt/PubState/fmt/json us/cm 49

Current behavior:

Unfortunately, the json_query = "measurements" configuration does not work together with json_time_key = "time".

Desired behavior:

It would be nice to be able to have the following:

[[inputs.mqtt_consumer]]
...
  data_format = "json"

  json_time_key = "time"
  json_time_format = "unix_ms"

  json_query = "measurements"

  tag_keys = [
    "name",
    "units"
  ]
...

to be able to extract time which is not part of measurements.

Use case:

Measurements formatted as:

{
    "time": 1555745371410,
    "measurements": [
        {
            "name": "temperature",
            "value": 23.4,
            "units": "℃"
        },
        {
            "name": "moisture",
            "value": 5,
            "units": "%"
        },
        {
            "name": "light",
            "value": 10118,
            "units": "lux"
        },
        {
            "name": "fertility",
            "value": 0,
            "units": "us/cm"
        }
    ]
}

allow to abstract away the measurements names and units. On the database level there will be no explicit measurement name notion (e.g. as a separate dedicated measurement), and no implicit units (which would be nice).

Regarding the time field placement - the following format might be also considered:

        "measurements": [
            {
                "name": "temperature",
                "value": 23.4,
                "units": "℃"
            },
            {
                "name": "moisture",
                "value": 5,
                "units": "%"
            },
            {
                "name": "light",
                "value": 10118,
                "units": "lux"
            },
            {
                "name": "fertility",
                "value": 0,
                "units": "us/cm"
            },
            {
                "name": "time",
                "value": 1555745371410,
                "units": "ms"
            }
        ]
    }

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/jsonjson and json_v2 parser/serialiser relatedfeature requestRequests for new plugin and for new features to existing plugins

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions