Skip to content

stability: consistency anomalies after RocksDB upgrade #19947

@tbg

Description

@tbg

We're seeing a lot of consistency violations on various clusters since RocksDB has been upgraded. For example, #19906 #19940 #19939 and a bunch of nightly failures are suspected to have this as a root cause.

From what I've seen, the consistency checker always finds that some replicas are missing big chunks of keyspace.

We should roll back the RocksDB upgrade to verify that this restores sanity, and then go searching for the origins of the problem. There is also the unlikely (and undesirable) case in which this is not actually related to the RocksDB bump, which we also want to assess quickly.

Metadata

Metadata

Assignees

Labels

S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions