Conversation
martinmr
left a comment
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @ashish-goswami and @manishrjain)
ashish-goswami
left a comment
There was a problem hiding this comment.
Please add a test if possible.
Reviewed 1 of 1 files at r1.
Reviewable status:
manishrjain
left a comment
There was a problem hiding this comment.
Also, do the priority as discussed.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @jarifibrahim, @manishrjain, and @martinmr)
levels.go, line 946 at r2 (raw file):
// computing compactability in order to guarantee progress. // Break the loop once L0 has enough space to accommodate new tables. if !s.isLevel0Compactable() {
If L0 not in memory, don't stall here.
The only stall should happen based on memory, not disk.
pawanrawal
left a comment
There was a problem hiding this comment.
Reviewable status: 0 of 3 files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @jarifibrahim, and @martinmr)
manishrjain
left a comment
There was a problem hiding this comment.
Reviewed 3 of 3 files at r3.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @jarifibrahim)
levels.go, line 426 at r3 (raw file):
prios = append(prios, pri) } }
We used to compact based on the score. But, we decided to compact based on the level, not the priority. So, upper levels always get compacted first, before the lower levels -- this allows us to avoid stalls.
Port dgraph-io#1186 Co-authored-by: Ibrahim Jarif <jarifibrahim@gmail.com>
Related to #1459 This PR contains the following changes to compactions - Use a separate thread for compacting Level 0 and 1 and a separate one for other levels - Pick levels to compact based on score. - Stall Level 0 if compactions cannot keep up (we had added this in #1186) - Limit the number of open table builders to 5 in compactions.
Related to #1459 This PR contains the following changes to compactions - Use a separate thread for compacting Level 0 and 1 and a separate one for other levels - Pick levels to compact based on score. - Stall Level 0 if compactions cannot keep up (we had added this in #1186) - Limit the number of open table builders to 5 in compactions. (cherry picked from commit 0b8eb4c)
Related to #1459 This PR contains the following changes to compactions - Use a separate thread for compacting Level 0 and 1 and a separate one for other levels - Pick levels to compact based on score. - Stall Level 0 if compactions cannot keep up (we had added this in #1186) - Limit the number of open table builders to 5 in compactions. (cherry picked from commit 0b8eb4c)
Related to #1459 This PR contains the following changes to compactions - Use a separate thread for compacting Level 0 and 1 and a separate one for other levels - Pick levels to compact based on score. - Stall Level 0 if compactions cannot keep up (we had added this in #1186) - Limit the number of open table builders to 5 in compactions.
Related to #1459 This PR contains the following changes to compactions - Use a separate thread for compacting Level 0 and 1 and a separate one for other levels - Pick levels to compact based on score. - Stall Level 0 if compactions cannot keep up (we had added this in #1186) - Limit the number of open table builders to 5 in compactions.
We don't need to stall writes if Level 1 does not have enough space. Level 1 is stored on the disk and it should be okay to have more number of tables (more size) on Level 1 than the
max level 1 size. These tables will eventually be compacted to lower levels.This commit changes the following
KeepL0InMemoryis true.This change is