[2.10] MOD-8097, MOD-8114 Fix memory counting (#5204)#5241
Merged
Conversation
* expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <guy.avimor@gmail.com> (cherry picked from commit 357d356)
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.10 #5241 +/- ##
=======================================
Coverage 86.48% 86.48%
=======================================
Files 192 192
Lines 34114 34128 +14
=======================================
+ Hits 29503 29517 +14
Misses 4611 4611 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
…fter removing the invecrted index
GuyAv46
approved these changes
Nov 21, 2024
|
|
||
| def check_index_info(env, idx, exp_num_records, exp_inv_idx_size, msg=""): | ||
| d = index_info(env, idx) | ||
| env.assertEqual(float(d['num_records']), exp_num_records, message=msg) |
Collaborator
There was a problem hiding this comment.
We should address the depth in all the assertions here. Without them we may miss the actual line in the test that caused the issue
Collaborator
Author
There was a problem hiding this comment.
This is the motivation behind passing msg
i added a message describing the location in the tests that call this function
Did you notice tests where it's missing or not informative enough?
raz-mon
added a commit
that referenced
this pull request
Jan 30, 2025
This reverts commit a97e925. Revert "[2.10] MOD-8601: Fix error message for LOAD (#5531) (#5550)" This reverts commit 5748233. Revert "[2.10] MOD-8561: Fix Inverted Index SeekTo Edge Case (#5536)" This reverts commit f9923b7. Revert "[2.10] Fix legacy geofilter leak - [MOD-8568] (#5516)" This reverts commit 8b75285. Revert "[2.10] MOD-7872: Add Types To Hide User Input (#5485)" This reverts commit d6acfed. Revert "[2.10] MOD-7864: Obfuscation API (#5487)" This reverts commit 259d417. Revert "MOD-7570 Backport 5422 to 2.10 (#5472)" This reverts commit e699dbc. Revert "[2.10] [MOD-8462] rename total_active_writes -> total_active_write_threads (#5446)" This reverts commit a90cfc3. Revert "[2.10] MOD-8058: Avoid error with _NUM_SSTRING (#5424)" This reverts commit 885cffb. Revert "[2.10] MOD-7634: Fix NOSTEM (#5392)" This reverts commit 4df508d. Revert "[2.10] Fixes for inverted indexes encoding - [MOD-8248] (#5388)" This reverts commit bc003c1. Revert "[2.10] Improve "Raw doc id" encoding - [MOD-8255] (#5372)" This reverts commit 9789e30. Revert "[2.10] Improve SkipToBlock logic - [MOD-8255] (#5370)" This reverts commit 56b16ea. Revert "[2.10] Improve counting active writes - [MOD-8151] (#5362)" This reverts commit a3d447d. Revert "[2.10] [MOD-8039] Reorganize stats files (#5356)" This reverts commit a1b8b29. Revert "[2.10] Improve text indexing performance - [MOD-8249] (#5352)" This reverts commit c5c9245. Revert "[2.10] [MOD-8115] Free spec resources in the main thread (#5327)" This reverts commit 42bb520. Revert "[2.10] Sorting Vector - [MOD-6783] (#5314)" This reverts commit 166a00e. Revert "[2.10] Fix long PREFIX/SUFFIX/INFIX/WILDCARD queries - [MOD-7882] (#5300)" This reverts commit 5c4ed74. Revert "[2.10] Remove assertion in optimizer (#5292)" This reverts commit ba55efb. Revert "[2.10] Fix missing expansions in text queries - [MOD-8142] (#5284)" This reverts commit 10f32d3. Revert "[2.10] Fix Tag OnReopen Callback - [MOD-8011] (#5282)" This reverts commit 2c029ee. Revert "[2.10] Fix MRIterator ownership mechanism - [MOD-8108] (#5266)" This reverts commit 457536e. Revert "[2.10] [MOD-8125] Create inverted index in write operations only (#5257)" This reverts commit 555557c. Revert "[2.10] [MOD-8152] rename and move new metrics (#5255)" This reverts commit a1f460e. Revert "[2.10] MOD-8129: Fix indexed_percent info stat (#5247)" This reverts commit d55afb5. Revert "[2.10] MOD-8097, MOD-8114 Fix memory counting (#5204) (#5241)" This reverts commit 52e58af. Revert "[MOD-7949] Avoid lazy expire upon scan keys in background [2.10] (#5294)" This reverts commit 6dfb503. Revert "[2.10] Cleanup for the trimming tree logic (#5226)" This reverts commit 7dfe6a2. Revert "[2.10] Add global marked deleted to stats [MOD-6053] (#5208)" This reverts commit f396732. Revert "Fix and add skips in pytest (#5210)" This reverts commit a8b505c. Revert "[2.10] [MOD-8035] Add field index error for JSON field indexing failures (#5202)" This reverts commit ed1919e. Revert "[2.10] Improve usage of RedisModuleCtx (#5181)" This reverts commit 8e686c7. Revert "[2.10] Add Active Indices Metrics - [MOD-7952] (#5179)" This reverts commit b72fc38. Revert "[2.10] MOD-8009: Allow Users To Configure Cursor Index Limitation Through Global Config (#5166)" This reverts commit 92e1dd8. Revert "[2.10] Send `SCORER` to the shards with `FT.AGGREGATE` on cluster mode - [MOD-8060] (#5163)" This reverts commit 4982ea2. Revert "[2.10] [MOD-8034] expose indexing failures to INFO modules (#5158)" This reverts commit d03f8de. Revert "[2.10] Add a counter of total query execution time - [MOD-7961] (#5153)" This reverts commit d45365d. Revert "[2.10] Add metrics of min and max memory consumption of indexes - [MOD-7584] (#5146)" This reverts commit 094318f. Revert "[2.10] Remove an unnecessary assertion - [MOD-8008] (#5142)" This reverts commit 3156f08. Revert "[2.10] Skip test_multithread:test_async_updates_sanity with sanitizer (#5125)" This reverts commit 5339033. Revert "Count Queries - [MOD-6492] (#5072)" This reverts commit 1006065. Revert "[2.10] Improve Error Message on VecSim Syntax Error - [MOD-7887] (#5109)" This reverts commit 9621fdc. Revert "[2.10] Fix flaky test (#5086)" This reverts commit 5aff118. Revert "[2.10] Fix flaky test_async_updates_sanity - avoid timeout in force invoke GC [MOD-7867] (#5081)" This reverts commit 2b85289. Revert "[2.10] MOD-7745: Add Benchmarks For Document And Field Expiration (#5063)" This reverts commit f99b0cf. Revert "[2.10] MOD-7863: Add HPEXPIRETIME Mock Support (#5028)" This reverts commit 79dfd0f.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backport #5204
conflicts:
Backport bug fix
This backport re-includes this bug fix.