etcdserver: compact raft log entries more frequently#18622
etcdserver: compact raft log entries more frequently#18622clement2026 wants to merge 1 commit intoetcd-io:mainfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: clement2026 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @clement2026. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Codecov Report✅ All modified and coverable lines are covered by tests. Please upload reports for the commit f1a76a2 to get more accurate results. Additional details and impacted files
... and 14 files with indirect coverage changes @@ Coverage Diff @@
## main #18622 +/- ##
==========================================
+ Coverage 68.79% 68.81% +0.01%
==========================================
Files 420 420
Lines 35522 35532 +10
==========================================
+ Hits 24438 24451 +13
+ Misses 9657 9654 -3
Partials 1427 1427 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Signed-off-by: Clement <gh.2lgqz@aleeas.com>
af9209f to
f1a76a2
Compare
| // Keep some in memory log entries for slow followers, while keeping the entries up to snapshot index. | ||
| // Only compact raft log once every N applies | ||
| if ep.appliedi <= ep.snapi+s.Cfg.SnapshotCatchUpEntries || ep.appliedi%s.Cfg.CompactRaftLogEveryNApplies != 0 { | ||
| return | ||
| } | ||
|
|
||
| compacti := ep.appliedi - s.Cfg.SnapshotCatchUpEntries |
There was a problem hiding this comment.
| // Keep some in memory log entries for slow followers, while keeping the entries up to snapshot index. | |
| // Only compact raft log once every N applies | |
| if ep.appliedi <= ep.snapi+s.Cfg.SnapshotCatchUpEntries || ep.appliedi%s.Cfg.CompactRaftLogEveryNApplies != 0 { | |
| return | |
| } | |
| compacti := ep.appliedi - s.Cfg.SnapshotCatchUpEntries | |
| // Retain all log entries up to the latest snapshot index to ensure any member can recover from that snapshot. | |
| // Beyond the snapshot index, preserve the most recent s.Cfg.SnapshotCatchUpEntries entries in memory. | |
| // This allows slow followers to catch up by synchronizing entries instead of requiring a full snapshot transfer. | |
| // Only compact raft log once every N applies | |
| if ep.snapi > s.Cfg.SnapshotCatchUpEntries { | |
| return | |
| } | |
| compacti := ep.snapi - s.Cfg.SnapshotCatchUpEntries |
In the first PR, let's keep the same number of entries, just make the compaction independent from snapshot.
|
This change does to many things, there are two changes here that should be independent:
|
|
Sure. I'll break down this PR. |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
Part of #17098
Goal
Compact raft log more frequently to save memory.
Key Changes
maybeCompactRaftLogto handle raft log compaction.CompactRaftLogEveryNAppliesandDefaultCompactRaftLogEveryNAppliesto control how frequently the raft log is compacted.TODO