Skip to content

Add support for getting recommended block_len/strong_len based on filesize. #129

@dbaarda

Description

@dbaarda

The current default block length and strongsum lengths are not ideal for best performance. The rsync binary includes a recommended block size of sqrt(file size), and a strong_sum length based on the block size and file size to give a compact signature without risking blocksum collisions. This makes it significantly faster for large files.

We should include these optimized defaults, baring in mind librsync doesn't have a whole-file checksum yet (rsync does) to avoid corruption from blocksum collisions, or a way to recover even if it did (rsync retries the transfer using different checksum seeds 3x before falling back to just copying the whole file). This means our strongsum length needs to be more conservative than rsync, particularly for protection against crafted attacks.

This requires extending the API to support providing the basis file size as a hint when generating the signature.

Note a signature filesize hint would also be nice for pre-allocating the signature blocksum data (the code currently doesn't expose a public API for this, but does have support under the hood).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions