remote write: reduce remote write egress bytes with new proto format#11999
remote write: reduce remote write egress bytes with new proto format#11999
Conversation
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
write request format Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
|
To extend this, we could allow the option of using alternative encoding/compression types. For example, with zstd we can achieve another 25% reduction in the end compressed bytes size but require double the amount of time to produce those compressed bytes: |
|
OTLP has done some experiments in this space and they saw a lot of benefit in compression and speed by trying an arrow based implementation. See https://github.com/open-telemetry/oteps/blob/main/text/0156-columnar-encoding.md and https://github.com/f5/otel-arrow-adapter/blob/main/docs/benchmarks.md I am not sure we should adopt it, but it might be something to evaluate. Further, my naive benchmark using avalanche data saw that OTLP (non-arrow) is more efficient on the compression. See: https://github.com/gouthamve/otlp-prw-compare |
|
The OTLP Arrow implementation has been accepted by the project on July 6th. Might be worth exploring more: https://docs.google.com/document/d/1-23Sf7-xZK3OL5Ogv2pK0NP9YotlSa0PKU9bvvtQwp8/edit#bookmark=id.fmr6zitz9egb |
|
Closing in favour of using a feature branch for remote write 1.1. This piece of work will be updated in a new PR. |
Design doc here
This PR is the result of conversations with multiple people over the last year about the remote write format and reuse of similar formats in other projects. I can't remember all the names but at the very least, thanks to: @rfratto @bboreham @cyriltovena @csmarchbanks
The short version is that this PR introduces a new _slightly different version of the remote write proto format that includes a table similar to TSDB's symbol table in each request, meaning rather than repeat many label name/value strings throughout the request we store them once and reference them via the table. While working on this I also came across an alternative library for snappy encoding, which compresses the request even more but is still uncompressable by the default Go snappy library.
The commits are sort of in order, no guarantees that each builds successfully.
The following benchmarks are of the building of the end compressed proto requests only, not including any excess memory or CPU required to process and cache with the new implementation prior to building the requests, or the decoding on the other end:
The current format but replacing the snappy library
The end bytes are smaller but building the request takes significantly longer to build.
The new format with the standard snappy library vs the current format and current snappy library
The new format with the alternate snappy library vs the current format and current snappy library
And finally the new format vs the new format with the alternate snappy library
These benchmarks are for relatively small synthetic datasets. While "Prometheus on my laptop" is not a production workload, it's a bit more realistic. The following screenshot is a comparison of the rate of compressed bytes sent by Prometheus, one using the new format and the other using the current format. Both Prometheus were running the same configs, scraping themselves. Over the course of the graph all that changes is the remote write config for the # of samples that can be batched into a remote write request:
