Rational

When emitting access logs in typed JSON format, certain fields can have null values, for example if the fields are populated from request headers or dynamic metadata. These fields are emitted to the log as JSON null.

This creates two similar problems:

• If the log destination is logstash, it doesn't support nullable indexed fields, so custom ruby filters need to be written, otherwise it would choke.
• In our infrastructure we ingest access logs into analytics pipelines. Nullable JSON fields create all sorts of problems when ingesting into strongly typed analytics DBs since whatever is doing the injection needs to explicitly define what to do with the nulls.

It is possible to remove the nulls as an intermediate step in a logging pipeline, but unfortunately logging agents like fluentd/fluent-bit and logstash have very poor support for null values in JSON logs, since nulls cannot be matched upon for filtering/manipulation.

To sum up, this can be worked around, but it would be much more convenient if envoy didn't emit null values at all.

Proposal
Introduce a ignore_empty_values boolean property defaulting to false in SubstitutionFormatString with the following semantics:

• For format_text - changes the output for empty operator values from - to "" so that empty values are skipped entirely.
• For format_json - removes the json output keys in case the value evaluated to null.

We will provide implementation/testing, seeking comments on the idea.