-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Implement better handling of OOD issues in optimizer #4297
Description
Is your feature request related to a problem? Please describe.
Currently, if the optimizer is used, it creates a new temporary segment with a copy of data from those segments, which are currently under optimization.
This process requires sometimes significant amount of disk space and if it is not enough, current implementation might panic.
Describe the solution you'd like
In optimization process, we can check if there are enough disk space to continue indexing or not.
A good space to do it is
| fn optimized_segment_builder( |
Describe alternatives you've considered
Indexed segment is potentially a plain structure. There are not too many places where we can't handle the OOD error.
Alternatively, we can check if we can actually handle OOD errors in this case, instead of checking for empty space each time.
Additional context
We created a test that simulates the scenario of disk overflow #4267
The solution for this issue should resolve this test scenario in the most general way reasonable.