Skip to content

Fluentd json parser to support multiple timestamp format for same set of logs #3248

@dontreboot

Description

@dontreboot

Is your feature request related to a problem? Please describe.

Fluentd json parser only supports a timestamp format in filter rule. I am using fluentd to parse Kubernetes docker logs and they have different timestamp formats. Some of them are in unix time format and others are in rfc3339 format. In order to support this use case, I have to explicitly define different filters and specify different time format for those json logs

Describe the solution you'd like
fluentd json plug that accepts multiple timestamp formats and use them to parse json logs. there are probably better ways to do it as that doesn't sound very efficient.

Describe alternatives you've considered
I use multiple filters to specify different timestamp formats.

<source>
  @type http
  port 9880
  bind 0.0.0.0
</source>

<filter vault.**>
  @type parser
  <parse>
    @type json
    json_parser json
    time_type string
    time_format "%Y-%m-%dT%H:%M:%S"
  </parse>
  key_name log
  hash_value_field json_log
  replace_invalid_sequence true
  emit_invalid_record_to_error false
  remove_key_name_field true
  reserve_data true
</filter>

<filter **>
  @type parser
  <parse>
    @type json
    json_parser json
  </parse>
  key_name log
  hash_value_field json_log
  replace_invalid_sequence true
  emit_invalid_record_to_error true
  remove_key_name_field true
  reserve_data true
</filter>

<match **>
  @type stdout
</match>

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementFeature request or improve operations

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions