Fixed by #7791

Aborting a stream records (or other) transfer may break future WAL delta transfers.

This PR adds a test to reproduce the problematic behavior.

Specifically, any new updates coming in during a stream records (or other) transfer may bump the last seen clocks. This also happens on the node that is receiving the transfer. If we abort the stream records transfer, the last seen clocks may have jumped over a huge gap. A follow up WAL delta transfer will only transfer changes since the last seen clocks, missing a huge set of changes in that jump.

In practice, this sequence is problematic:

• start stream records transfer to recover replica
• send new update through all peers
• abort stream records transfer
• start WAL delta transfer
• ⚠️ WAL delta resolves very short or empty diff

Imagine the initial stream records transfer only covered 10% of the points before it got aborted. The WAL delta transfer after it will then miss the remaining 90% of point changes, corrupting the target replica.

Test:

cargo build --features staging
pytest tests/consensus_tests/test_shard_wal_delta_transfer.py -k test_abort_stream_records_breaks_wal_delta

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you successfully ran tests with your changes locally?