-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: optimize new SST iterators #83051
Copy link
Copy link
Closed
cockroachdb/pebble
#1905Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.A-storageRelating to our storage engine (Pebble) on-disk storage.Relating to our storage engine (Pebble) on-disk storage.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)C-performancePerf of queries or internals. Solution not expected to change functional behavior.Perf of queries or internals. Solution not expected to change functional behavior.T-storageStorage TeamStorage Team
Description
The new SST iterators in #82799, e.g. NewPebbleSSTMemIterator(), are much slower than the existing NewMemSSTIterator(), around 80% or so. Some of this is due to CRDB-side overhead, in particular the use of pebbleIterator and VerifyingMVCCIterator, but the majority appears to be in Pebble via the new NewExternalIter() SST iterator. We need to optimize these.
This can be seen with BenchmarkSSTIterator, by changing NewMemSSTIterator() to NewPebbleMemSSTIterator() in runSSTIterator() on top of #82799.
Many of the optimizations that address #82559 and #83049 would likely apply here as well.
Jira issue: CRDB-16822
Epic CRDB-2624
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.A-storageRelating to our storage engine (Pebble) on-disk storage.Relating to our storage engine (Pebble) on-disk storage.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)C-performancePerf of queries or internals. Solution not expected to change functional behavior.Perf of queries or internals. Solution not expected to change functional behavior.T-storageStorage TeamStorage Team