-
Notifications
You must be signed in to change notification settings - Fork 4.1k
changefeed: Increase maximum message sizes. #76258
Description
Sarama library used to write to kafka restricts a message to be at most
MaxMessageBytes bytes -- which is set to 1MB.
In addition, sarama imposes a limit of MaxRequestSize for a request (100MB).
Those 2 settings are currently not configurable, and we have customers who start storing very large
(5+MB) JSON blobs that can easily exceed MaxMessageBytes.
We need to make changes to support those messages. One way to do this would be to extend
kafka_sink_config to support those options. But perhaps, a better approach is to just make those
"unlimited". We already limit amount of memory that can be consumed by changefeed, and thus
if we were able to ingest the message, and convert it to appropriate format (avro or json), then perhaps
the library itself should not apply any additional limits and any failures should come from the downstream
rejecting those messages.
Epic CRDB-8665