mvcc: gc write CF with compaction filter#5333
Conversation
e73e42f to
a63afc6
Compare
a20612f to
d8bc119
Compare
|
It can be reviewd now. PTAL @MyonKeminta @Little-Wallace thanks! |
src/raftstore/store/fsm/apply.rs
Outdated
There was a problem hiding this comment.
I don't think it's a good idea to leak txn concepts to raft layer. How about inspecting the deleted keys and sending MvccGcTask in raftstore::coprocessor?
There was a problem hiding this comment.
I think it's not raft layer, it's just a proposal that needs to do some special things for it. We also have ComputeHash like this.
cd9cae6 to
d176c7c
Compare
|
PTAL @MyonKeminta thanks! |
There was a problem hiding this comment.
Create thread for every task? why not threadpool?
There was a problem hiding this comment.
Emm, we have removed many thread pools. I'm afraid to add one more. what do you think @zhangjinpeng1987 ?
There was a problem hiding this comment.
I think we could make Rocksdb support CompactRange in async way, which just put it into background threadpool of Rocksdb. @yiwu-arbug
There was a problem hiding this comment.
Good idea. So let's keep the current implementation, and switch to the async API when it's ready.
There was a problem hiding this comment.
keep the current way and leave a TODO comment here?
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
14c91d2 to
ea2a791
Compare
Signed-off-by: qupeng <qupeng@pingcap.com>
|
|
||
| use crate::raftstore::store::keys::{data_end_key, data_key}; | ||
| use crate::storage::mvcc::{ | ||
| extract_mvcc_props, get_gc_ratio, init_mvcc_gc_db, init_safe_point, need_gc, |
There was a problem hiding this comment.
I think you'd better reconsider the overall architecture to avoid coupling between raftstore and storage.
| let dek = data_end_key(&task.end_key); | ||
| if *sk < dek && dsk < *ek { | ||
| // overlap with a range in compaction. | ||
| info!("gc task overlap, skip"); |
There was a problem hiding this comment.
Why not use a BTreeMap instead of a HashMap, so you don't need to iterate all ranges and check all of them?
| } | ||
|
|
||
| pub fn need_gc(props: MvccProperties, safe_point: u64, ratio_threshold: f64) -> bool { | ||
| if ratio_threshold <= 1.0 { |
There was a problem hiding this comment.
This seems to be duplicated with Reader::need_gc. I think you can invoke this function in Reader::need_gc.
What have you changed?
GC write CF with compaction filter.
What is the type of the changes?
Improvement.
How is the PR tested?
Unit tests
Benchmark result if necessary (optional)
With the patch:
Without the patch:
Dependencies