Skip to content

changefeed: Increase maximum message sizes. #76258

@miretskiy

Description

@miretskiy

Sarama library used to write to kafka restricts a message to be at most
MaxMessageBytes bytes -- which is set to 1MB.
In addition, sarama imposes a limit of MaxRequestSize for a request (100MB).

Those 2 settings are currently not configurable, and we have customers who start storing very large
(5+MB) JSON blobs that can easily exceed MaxMessageBytes.

We need to make changes to support those messages. One way to do this would be to extend
kafka_sink_config to support those options. But perhaps, a better approach is to just make those
"unlimited". We already limit amount of memory that can be consumed by changefeed, and thus
if we were able to ingest the message, and convert it to appropriate format (avro or json), then perhaps
the library itself should not apply any additional limits and any failures should come from the downstream
rejecting those messages.

Epic CRDB-8665

Metadata

Metadata

Assignees

Labels

A-cdcChange Data CaptureC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-cdc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions