Skip to content

Implement better handling of OOD issues in optimizer #4297

@generall

Description

@generall

Is your feature request related to a problem? Please describe.

Currently, if the optimizer is used, it creates a new temporary segment with a copy of data from those segments, which are currently under optimization.
This process requires sometimes significant amount of disk space and if it is not enough, current implementation might panic.

Describe the solution you'd like

In optimization process, we can check if there are enough disk space to continue indexing or not.
A good space to do it is

Describe alternatives you've considered
Indexed segment is potentially a plain structure. There are not too many places where we can't handle the OOD error.
Alternatively, we can check if we can actually handle OOD errors in this case, instead of checking for empty space each time.

Additional context

We created a test that simulates the scenario of disk overflow #4267

The solution for this issue should resolve this test scenario in the most general way reasonable.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions