Skip to content

Draft: flush segments in two stages#7443

Closed
timvisee wants to merge 14 commits intodevfrom
segments-flush-two-stages
Closed

Draft: flush segments in two stages#7443
timvisee wants to merge 14 commits intodevfrom
segments-flush-two-stages

Conversation

@timvisee
Copy link
Member

@timvisee timvisee commented Oct 23, 2025

WIP

Flush all segments in two stages:

  1. inserts and updates
  2. deletes

This is a naive implementation that allows us to test this idea. If it works out, we can polish it further.

We've seen a variety of issues where points did go missing. Almost all of them are related to us deleting them on disk too early. Segments have a strong dependency with each other in terms of flushing order to ensure data consistency.

The rough idea of this change is to always flush deletes as very last step. This will hopefully prevent all such issues. We rely on the WAL to repair all cases where we might have partially flushed.

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

@timvisee timvisee mentioned this pull request Oct 23, 2025
6 tasks
@timvisee timvisee force-pushed the segments-flush-two-stages branch 3 times, most recently from 1ab7eef to 9252f68 Compare October 27, 2025 14:38
@timvisee timvisee changed the base branch from dev to gridstore-flush-closure October 27, 2025 16:15
@timvisee timvisee force-pushed the gridstore-flush-closure branch from 575ec80 to d482fa7 Compare November 18, 2025 12:30
Base automatically changed from gridstore-flush-closure to dev November 18, 2025 13:21
@timvisee timvisee force-pushed the segments-flush-two-stages branch 2 times, most recently from 6f9a5c6 to 8299a46 Compare December 5, 2025 15:47
@timvisee timvisee force-pushed the segments-flush-two-stages branch from 8299a46 to fdb75b8 Compare December 12, 2025 12:16
@timvisee
Copy link
Member Author

Consistency has improved significantly in recent versions, especially with merging #7850.

Closing this as I don't think the more complicated flushing logic will give us benefit.

@timvisee timvisee closed this Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant