You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The following error is encountered when testing V2 table.
Page checksum mismatch, actual=3372108349 vs expect=3375987274
The reasons are that HashUtil::crc_hash doesn't generate standard CRC-32 checksum and produces wrong checksum on vector of Slice.
I have changed V2 to use CRC-32C checksum, which is the de factor file checksum algorithm nowadays that is being used in storage systems like RocksDB, Ceph, etc.
Even if HashUtils::crc_hash doesn't generate standard CRC-32 checksum, the checksums should be equal. Because all the checksums are compute by this function.
we have already had a CRC-32 implementation in be/src/olap/utils.cpp. Can you unify this and your new implementation?
Even if HashUtils::crc_hash doesn't generate standard CRC-32 checksum, the checksums should be equal. Because all the checksums are compute by this function.
The real problem is that the result of crc_hash can't be used as the input of the next crc_hash call. That is, crc_hash({"hello ", "world"}, 0) != crc_hash({"hello"}, 0).
we have already had a CRC-32 implementation in be/src/olap/utils.cpp. Can you unify this and your new implementation?
olap_adler32 and olap_crc32 are both used in V1 segment, which we can't change because of compatibility issue. I think they can be removed when we're removing V1 related code.
Even if HashUtils::crc_hash doesn't generate standard CRC-32 checksum, the checksums should be equal. Because all the checksums are compute by this function.
The real problem is that the result of crc_hash can't be used as the input of the next crc_hash call. That is, crc_hash({"hello ", "world"}, 0) != crc_hash({"hello"}, 0).
we have already had a CRC-32 implementation in be/src/olap/utils.cpp. Can you unify this and your new implementation?
olap_adler32 and olap_crc32 are both used in V1 segment, which we can't change because of compatibility issue. I think they can be removed when we're removing V1 related code.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The following error is encountered when testing V2 table.
The reasons are that
HashUtil::crc_hashdoesn't generate standard CRC-32 checksum and produces wrong checksum on vector of Slice.I have changed V2 to use CRC-32C checksum, which is the de factor file checksum algorithm nowadays that is being used in storage systems like RocksDB, Ceph, etc.