-
Notifications
You must be signed in to change notification settings - Fork 4.1k
stability: consistency anomalies after RocksDB upgrade #19947
Copy link
Copy link
Closed
Labels
S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restartingSevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting
Description
We're seeing a lot of consistency violations on various clusters since RocksDB has been upgraded. For example, #19906 #19940 #19939 and a bunch of nightly failures are suspected to have this as a root cause.
From what I've seen, the consistency checker always finds that some replicas are missing big chunks of keyspace.
We should roll back the RocksDB upgrade to verify that this restores sanity, and then go searching for the origins of the problem. There is also the unlikely (and undesirable) case in which this is not actually related to the RocksDB bump, which we also want to assess quickly.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restartingSevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting