[2.8] MOD-8097, MOD-8114 Fix memory counting#5242
Merged
Conversation
* expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <guy.avimor@gmail.com> (cherry picked from commit 357d356)
GuyAv46
approved these changes
Nov 21, 2024
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.8 #5242 +/- ##
=======================================
Coverage 84.71% 84.72%
=======================================
Files 195 195
Lines 34149 34161 +12
=======================================
+ Hits 28931 28943 +12
Misses 5218 5218 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
kei-nan
added a commit
that referenced
this pull request
Jan 30, 2025
This reverts commit ef25dca. Revert "[2.8] Fix Max Frequency Misscalculation - [MOD-8158] (#5560)" This reverts commit a6864e3. Revert "[2.8] MOD-8561: Fix Inverted Index SeekTo Edge Case (#5542)" This reverts commit bb873af. Revert "[2.8] Fix legacy geofilter leak - [MOD-8568] (#5518)" This reverts commit 979f3d2. Revert "Backport 4243 to 2.8 (#5508)" This reverts commit 9af7959. Revert "MOD-7570 Backport 5422 to 2.8 (#5470)" This reverts commit 540b5f3. Revert "[2.8] MOD-7872: Add Types To Hide User Input (#5489)" This reverts commit b2bf242. Revert "[2.8] MOD-7864: Obfuscation API (#5488)" This reverts commit 4866bf1. Revert "[2.8] MOD-7634: Fix NOSTEM (#5475)" This reverts commit a015a55. Revert "[2.8] [MOD-8462] rename total_active_writes -> total_active_write_threads (#5444)" This reverts commit baadb4d. Revert "[2.8] MOD-8058: Avoid error with _NUM_SSTRING (#5187) (#5427)" This reverts commit 5b49276. Revert "Backport metrics info 2.8 [MOD-8388] (#5407)" This reverts commit f428033. Revert " [2.8] Fixes for inverted indexes encoding - [MOD-8248] (#5389)" This reverts commit 0c7227e. Revert "[2.8] MOD-8221: backport info fields (for metrics) to 2.8 (#5358)" This reverts commit b7eaa56. Revert "[2.8] Improve "Raw doc id" encoding - [MOD-8255] (#5371)" This reverts commit 152043b. Revert "[2.8] Improve SkipToBlock logic - [MOD-8255] (#5369)" This reverts commit c1a2c8f. Revert "[2.8][MOD-8115] Free spec resources in the main thread (#5324) (#5330)" This reverts commit 5f3ce64. Revert "[2.8] Fix Sorting Vector - [MOD-6783] (#5316)" This reverts commit 04f7778. Revert "[2.8] Fix a flakiness in testCursorOnCoordinator - [MOD-8016] (#5305)" This reverts commit 4fc1b27. Revert "[MOD-7949] Avoid lazy expire upon scan keys in background [2.8] (#5303)" This reverts commit 16e4ff4. Revert "[2.8] Fix long PREFIX/SUFFIX/INFIX/WILDCARD queries - [MOD-7882] (#5299)" This reverts commit 09f5b62. Revert "[2.8] Remove assertion in optimizer (#5296)" This reverts commit 45db16a. Revert "[2.8] MOD-8188: Fix OnReopen Comments (#5288)" This reverts commit 001746f. Revert "[2.8] Fix missing expansions in text queries - [MOD-8142] (#5287)" This reverts commit ed13507. Revert "[2.8] Fix Tag OnReopen Callback - [MOD-8011] (#5280)" This reverts commit 40b8012. Revert "[2.8] Fix MRIterator ownership mechanism - [MOD-8108] (#5269)" This reverts commit 90a5685. Revert "[2.8] [MOD-8125] Create inverted index in write operations only (#5260)" This reverts commit ebb9484. Revert "[2.8] MOD-8129: Fix indexed_percent info stat (#5249)" This reverts commit fb9781a. Revert "[2.8] MOD-8097, MOD-8114 Fix memory counting (#5242)" This reverts commit a21a909. Revert "[2.8] Cleanup for the trimming tree logic (#5225)" This reverts commit 1b9126b. Revert "[2.8] [MOD-8035] Add field index error for JSON field indexing failures (#5205)" This reverts commit 3a13ddb. Revert "[2.8] Fix Numeric Tree Balance - [MOD-8081, MOD-8082] (#5200)" This reverts commit a2948c4. Revert "[2.8] Performance improvement in indexBulkFields - [MOD-8093] (#5192)" This reverts commit 2b0e0a0. Revert "[2.8] MOD-8009: Allow Users To Configure Cursor Index Limitation Through Global Config (#5167)" This reverts commit f860505. Revert "[2.8] Skip test_multithread:test_async_updates_sanity with sanitizer (#5124)" This reverts commit 938369a. Revert "[2.8] Improve Error Message on VecSim Syntax Error - [MOD-7887] (#5108)" This reverts commit ba614ca. Revert "[2.8] Fix flaky test (#5084)" This reverts commit f579156. Revert "[2.8] Fix flaky test_async_updates_sanity - avoid timeout in force invoke GC [MOD-7867] (#5080)" This reverts commit 1a0eabc. Revert "[2.8] Handle VecSim in-place deletion after async [MOD-7643, MOD-7732] (#5048)" This reverts commit fcc19e9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backport #5204
conflicts: