We're seeing a lot of consistency violations on various clusters since RocksDB has been upgraded. For example, #19906 #19940 #19939 and a bunch of nightly failures are suspected to have this as a root cause.
From what I've seen, the consistency checker always finds that some replicas are missing big chunks of keyspace.
We should roll back the RocksDB upgrade to verify that this restores sanity, and then go searching for the origins of the problem. There is also the unlikely (and undesirable) case in which this is not actually related to the RocksDB bump, which we also want to assess quickly.
We're seeing a lot of consistency violations on various clusters since RocksDB has been upgraded. For example, #19906 #19940 #19939 and a bunch of nightly failures are suspected to have this as a root cause.
From what I've seen, the consistency checker always finds that some replicas are missing big chunks of keyspace.
We should roll back the RocksDB upgrade to verify that this restores sanity, and then go searching for the origins of the problem. There is also the unlikely (and undesirable) case in which this is not actually related to the RocksDB bump, which we also want to assess quickly.