Currently we default to 8mb chunks in our uploads to cloud storage. When running many (e.g. 6 workers) per backup, and potentially concurrent backups, with each worker sitting on top of an SDK that may have chunks in progress or in queues, these larger chunks use more memory and can take longer to process, for example to hash a single chunk (see #115192 and #115189).
A smaller chunk size would slightly increase the number of parts we upload but for most of our ~128mb files, going to 5mb would mean something like 25 part uploads instead of 16. This seems minor compared to the memory and hashing benefits.
Note: s3 has a minimum chunk size of 5mb, so we only have room to cut the current 8mib default by 3/8th. But that still may be worth doing.
Jira issue: CRDB-33927
Currently we default to 8mb chunks in our uploads to cloud storage. When running many (e.g. 6 workers) per backup, and potentially concurrent backups, with each worker sitting on top of an SDK that may have chunks in progress or in queues, these larger chunks use more memory and can take longer to process, for example to hash a single chunk (see #115192 and #115189).
A smaller chunk size would slightly increase the number of parts we upload but for most of our ~128mb files, going to 5mb would mean something like 25 part uploads instead of 16. This seems minor compared to the memory and hashing benefits.
Note: s3 has a minimum chunk size of 5mb, so we only have room to cut the current 8mib default by 3/8th. But that still may be worth doing.
Jira issue: CRDB-33927