Skip to content

Add test for OOD during indexing#4267

Merged
generall merged 4 commits intodevfrom
test/low-disk-indexing
Jun 29, 2024
Merged

Add test for OOD during indexing#4267
generall merged 4 commits intodevfrom
test/low-disk-indexing

Conversation

@tellet-q
Copy link
Copy Markdown
Contributor

@tellet-q tellet-q commented May 20, 2024

Automate a test scenario for OutOfDisk error during indexing.

Scenario:
Limit Qdrant container disk to ~10 MB

  1. create collection with "indexing_threshold": 0 (disabled indexing)
  2. add points
  3. enable indexing
  4. wait for indexing
  5. send a search request

Expected behavior: indexing should gracefully fail without crushing a container and post a log message, the container should continue running and accept search requests.

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

@tellet-q tellet-q changed the title Add test for OOD during indexing WIP: Add test for OOD during indexing May 20, 2024
@tellet-q tellet-q force-pushed the test/low-disk-indexing branch from efdca9a to 780ed2d Compare May 21, 2024 07:24
@tellet-q tellet-q requested review from generall and timvisee May 21, 2024 07:45
@tellet-q tellet-q changed the title WIP: Add test for OOD during indexing Add test for OOD during indexing May 21, 2024
@generall generall force-pushed the test/low-disk-indexing branch from e35c780 to a0f6c4d Compare June 28, 2024 08:29
* fs4@0.8.4

* fail early on low storage

* move `dir_size` to `common`

* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`

* drop dead code

* move dir_size to common subcrate

---------

Co-authored-by: generall <andrey@vasnetsov.com>
@generall generall merged commit c88f10c into dev Jun 29, 2024
@generall generall deleted the test/low-disk-indexing branch June 29, 2024 20:40
generall added a commit that referenced this pull request Jun 29, 2024
* Add test for OOD during indexing

* Start indexing right after 1st OOD message

* Only send search request after insert loop

* Fail early when encountering out-of-storage during optimization (#4578)

* fs4@0.8.4

* fail early on low storage

* move `dir_size` to `common`

* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`

* drop dead code

* move dir_size to common subcrate

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants