Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode #3567
Description
BUG REPORT
Describe the bug
Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode
To Reproduce
Steps to reproduce the behavior:
- Run with high workload with openmessaging-benchmark(2000MB/s)
- 3 bookies
- 3 benchmark-worker
- 3 replica
- 3 writeQuorumSize
- 2 ackQuorumSize
- See error
Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:49 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Expected behavior
Caused by: org.rocksdb.RocksDBException: block checksum mismatch
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.