Part of #115190
We have observed that backups to s3 cause increased tail latencies in foreground traffic, sometimes significantly.
The S3 SDK hashes chunk sized (currently 8mb) blocks both with MD5 and SHA256, for content checksum and signing respectively. It appears that due in large part to golang/go#64417, this causes us to observe long gc pause times and traces show STW pauses overlapping with block hashing.
We can disable the MD5 hashing as it is also done by amazon server side, and thus exists only to verify the upload integrity, but this is already verified by uploading over TLS, while additionally most of the files we upload already have internal or other hashes included in their content, such as SST block checksums.
Jira issue: CRDB-33923
Part of #115190
We have observed that backups to s3 cause increased tail latencies in foreground traffic, sometimes significantly.
The S3 SDK hashes chunk sized (currently 8mb) blocks both with MD5 and SHA256, for content checksum and signing respectively. It appears that due in large part to golang/go#64417, this causes us to observe long gc pause times and traces show STW pauses overlapping with block hashing.
We can disable the MD5 hashing as it is also done by amazon server side, and thus exists only to verify the upload integrity, but this is already verified by uploading over TLS, while additionally most of the files we upload already have internal or other hashes included in their content, such as SST block checksums.
Jira issue: CRDB-33923