feat: implement application-level compression for consumer-provider communication#2101
Merged
Conversation
187bb81 to
d017e35
Compare
Test Results2 994 tests +37 2 993 ✅ +37 23m 9s ⏱️ - 10m 10s Results for commit 0cc3a80. ± Comparison against base commit af7b939. This pull request removes 1 and adds 38 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
d017e35 to
a9cd8f3
Compare
avitenzer
previously approved these changes
Nov 20, 2025
…ommunication - Add custom compression logic in protocol/common/compression.go - CompressData: compress payloads > 10KB using gzip.BestSpeed - DecompressData: decompress gzip data - Size-based logic: only compress if it reduces size - Implement custom header system to bypass grpc-web wrapper issues - lava-compression-support: consumer signals compression capability - lava-compression: provider indicates compressed response - Replaces reliance on grpc-accept-encoding (always present) - Consumer side (protocol/rpcconsumer/rpcconsumer_server.go) - Send lava-compression-support header when flag enabled - Detect and decompress responses with lava-compression header - Handle decompression errors gracefully - Provider side (protocol/rpcprovider/rpcprovider_server.go) - Check lava-compression-support header from consumer - Compress responses > 10KB when consumer supports it - Set lava-compression header on compressed responses - Skip compression if it doesn't reduce size - Update provider listener with compression support info This implementation works around the grpc-web wrapper stripping native gRPC compression headers (grpc-encoding), enabling compression for large payloads (15-45 MB) to reduce bandwidth usage by 70-85%. Controlled by existing flag: --allow-grpc-compression-for-consumer-provider-communication
Unit Tests (compression_test.go): - Test CompressData with data below/at/above threshold - Test CompressData with incompressible data - Test DecompressData with valid and invalid gzip data - Test full compression/decompression round-trip cycles - Test edge cases (exact threshold, one byte above threshold) - Test large payloads similar to real RPC responses (15MB) - Add benchmarks for different payload sizes (10KB, 100KB, 1MB) - Achieves 99.54% compression on large test payloads Integration Tests (compression_integration_test.go): - TestCompressionHeadersConsumerToProvider: Verify consumer sets compression support header - TestCompressionHeadersProviderToConsumer: Verify provider sets compression header - TestConsumerDecompressionFlow: Test complete consumer decompression logic - TestProviderCompressionDecisionFlow: Test provider's decision logic - TestEndToEndCompressionFlow: Full integration test simulating consumer->provider->consumer - With compression enabled (99.3% compression ratio) - Without compression support - TestCompressionWithRealGzipData: Verify compatibility with standard gzip - TestCompressionWithIncompressibleData: Test behavior with random data Total: 21 test cases + 6 benchmarks, all passing
0cc3a80 to
454ed78
Compare
454ed78 to
5e6be70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add custom compression logic in protocol/common/compression.go
Implement custom header system to bypass grpc-web wrapper issues
Consumer side (protocol/rpcconsumer/rpcconsumer_server.go)
Provider side (protocol/rpcprovider/rpcprovider_server.go)
Update provider listener with compression support info
This implementation works around the grpc-web wrapper stripping native gRPC compression headers (grpc-encoding), enabling compression for large payloads (15-45 MB) to reduce bandwidth usage by 70-85%.
Controlled by existing flag:
--allow-grpc-compression-for-consumer-provider-communication
Description
Closes: #XXXX
Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
!in the type prefix if API or client breaking changemainbranchReviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...