fix(hash): Untrack hash with volatile fields when it is overwritten #3003
Conversation
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
18d6f28 to
1ce65a8
Compare
|
Ain't gonna lie I am not sure about this test here! Is this related to this or is this something else? |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #3003 +/- ##
============================================
- Coverage 74.29% 74.02% -0.27%
============================================
Files 129 129
Lines 70895 70899 +4
============================================
- Hits 52670 52485 -185
- Misses 18225 18414 +189
🚀 New features to boost your workflow:
|
not related to #3000. this is a "day 1" HFE bug |
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
dvkashapov
left a comment
There was a problem hiding this comment.
I'm not sure initial suggestion is sufficient but I may be wrong. In case where both old and new are hashes with volatile fields, shouldn't pointer be updated if it changed?
| serverAssert(expireref != NULL); | ||
| *expireref = new; | ||
| } | ||
| } |
There was a problem hiding this comment.
@ranshid After this line, shouldn't we do something like this?
if (old_was_hash_with_volatile) {
bool new_is_hash_with_volatile = (new->type == OBJ_HASH && hashTypeHasVolatileFields(new));
if (new_is_hash_with_volatile) {
int dict_index = getKVStoreIndexForKey(key->ptr);
hashtable *volatile_items_ht = kvstoreGetHashtable(db->keys_with_volatile_items, dict_index);
hashtableReplaceReallocatedEntry(volatile_items_ht, old, new);
} else {
dbUntrackKeyWithVolatileItems(db, new);
}
} else {
dbTrackKeyWithVolatileItems(db, new);
}where old_was_hash_with_volatile = (old->type == OBJ_HASH && hashTypeHasVolatileFields(old));
There was a problem hiding this comment.
So theoretically if there is a case were someone will swap the content of a hash object with a different hash object I would say yes. the thing is, I do not think there is anyway this is currently done today.
The thing is that dbSetValue is called from setKey, dbAddInternal and dbReplaceValue
in all cases were the the old value was hash, it will be untracked (as we wanted).
The problem you mention is "what happen in case the new value is also a hash with volatile items?"
I think there is no case currently which allows that, but I would say that we can add a generic support for that by just calling
dbTrackKeyWithVolatileItems(db, new);
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
|
@frostzt lets take @dvkashapov advise and make the change instead like: |
@ranshid Awesome just so I get the idea right here, the It means that when we do We can't really create |
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
| dbUntrackKeyWithVolatileItems(db, old); | ||
| } | ||
| /* If the new object is a hash with volatile items we need to track it again */ | ||
| dbTrackKeyWithVolatileItems(db, new); |
There was a problem hiding this comment.
oh yeah we can't do this here its uninitialized here!
There was a problem hiding this comment.
@ranshid should we move this dbTrackKeyWithVolatileItems inside the else...if (expiry... block?
There was a problem hiding this comment.
no you should move this entire new section to line 379 (373 in the original implementation)
I think there is no easy way you can currently test the case of "in-place replacing a hash with a different hash" |
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
| /* If overwriting a hash object, un-track it from the volatile items tracking if it contains volatile items.*/ | ||
| if (old->type == OBJ_HASH && hashTypeHasVolatileFields(old)) { | ||
| dbUntrackKeyWithVolatileItems(db, old); | ||
| } |
There was a problem hiding this comment.
this section should be moved before the section you added in line 378
There was a problem hiding this comment.
ok now it makes total sense to me, actually when we put this part inside the overwrite I was a little confused as the comment above says that its only when we perform edits!
Signed-off-by: frostzt <aidenfrostbite@gmail.com>
ranshid
left a comment
There was a problem hiding this comment.
Perfect! Thank you @frostzt and @dvkashapov !
Thanks a lot @ranshid honestly I am more surprised by how much you know this thing internally! Please feel free to mention me to issues or (if you think) features that you feel I can pick up! I am more than eager to contribute to Valkey 😃 Hopefully I can clear our your backlogs maybe 🤣 |
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in #3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
…3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check #3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in #3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
…alkey-io#3003) In `dbSetValue` if the robj is a hash with volatile fields we need to clean it up. It causes two core problems: 1. Type Confusion: Allows converting from hash to string with `keys_with_volatile_items` tracked 2. Tracked Items with dangling pointer: Converting from `hash` to ie. a string breaks this as `keys_with_volatile_items` is not cleaned up. Check valkey-io#3000 --------- Signed-off-by: frostzt <aidenfrostbite@gmail.com> Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
In `dbSetValue()` the `old` pointer may be reassigned to point to the incoming value object which was created without an embedded key, so calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()` which returns NULL, causing a crash in `hashtableSdsHash()` when trying to hash the NULL key. Idea is to assign `old_was_hash_with_volatile` before the swap and use `new` instead of `old` for untracking when theres no embedded key. Introduced in valkey-io#3003 Run with NULL ptr dereference: https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880 --------- Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com> Co-authored-by: Ran Shidlansik <ranshid@amazon.com> Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [docker.io/valkey/valkey](https://github.com/valkey-io/valkey) | image | patch | `9.0.1` → `9.0.4` | --- ### Release Notes <details> <summary>valkey-io/valkey (docker.io/valkey/valkey)</summary> ### [`v9.0.4`](https://github.com/valkey-io/valkey/releases/tag/9.0.4) [Compare Source](valkey-io/valkey@9.0.3...9.0.4) Upgrade urgency SECURITY: This release includes security fixes we recommend you apply as soon as possible. ##### Security fixes - (CVE-2026-23479) Use-After-Free in unblock client flow - (CVE-2026-25243) Invalid Memory Access in RESTORE command - (CVE-2026-23631) Use-after-free when full sync occurs during a yielding Lua/function execution ### [`v9.0.3`](https://github.com/valkey-io/valkey/releases/tag/9.0.3) [Compare Source](valkey-io/valkey@9.0.2...9.0.3) ##### Valkey 9.0.3 Upgrade urgency SECURITY: This release includes security fixes we recommend you apply as soon as possible. ##### Security fixes - (CVE-2025-67733) RESP Protocol Injection via Lua error\_reply - (CVE-2026-21863) Remote DoS with malformed Valkey Cluster bus message - (CVE-2026-27623) Reset request type after handling empty requests ##### Bug fixes - Avoids crash during MODULE UNLOAD when ACL rules reference a module command and subcommand ([#​3160](valkey-io/valkey#3160)) - Fix server assert on ACL LOAD when current user loses permission to channels ([#​3182](valkey-io/valkey#3182)) - Fix bug causing no response flush sometimes when IO threads are busy ([#​3205](valkey-io/valkey#3205)) ### [`v9.0.2`](https://github.com/valkey-io/valkey/releases/tag/9.0.2) [Compare Source](valkey-io/valkey@9.0.1...9.0.2) Upgrade urgency HIGH: There are critical bugs that may affect a subset of users. #### Bug fixes - Avoid memory leak of new argv when HEXPIRE commands target only non-exiting fields ([#​2973](valkey-io/valkey#2973)) - Fix HINCRBY and HINCRBYFLOAT to update volatile key tracking ([#​2974](valkey-io/valkey#2974)) - Avoid empty hash object when HSETEX added no fields ([#​2998](valkey-io/valkey#2998)) - Fix case-sensitive check for the FNX and FXX arguments in HSETEX ([#​3000](valkey-io/valkey#3000)) - Prevent assertion in active expiration job after a hash with volatile fields is overwritten ([#​3003](valkey-io/valkey#3003), [#​3007](valkey-io/valkey#3007)) - Fix HRANDFIELD to return null response when no field could be found ([#​3022](valkey-io/valkey#3022)) - Fix HEXPIRE to not delete items when validation rules fail and expiration is in the past ([#​3023](valkey-io/valkey#3023), [#​3048](valkey-io/valkey#3048)) - Fix how hash is handling overriding of expired fields overwrite ([#​3060](valkey-io/valkey#3060)) - HSETEX - Always issue keyspace notifications after validation ([#​3001](valkey-io/valkey#3001)) - Make zero a valid TTL for hash fields during import mode and data loading ([#​3006](valkey-io/valkey#3006)) - Trigger prepareCommand on argc change in module command filters ([#​2945](valkey-io/valkey#2945)) - Restrict TTL from being negative and avoid crash in import-mode ([#​2944](valkey-io/valkey#2944)) - Fix chained replica crash when doing dual channel replication ([#​2983](valkey-io/valkey#2983)) - Skip slot cache optimization for AOF client to prevent key duplication and data corruption ([#​3004](valkey-io/valkey#3004)) - Fix used\_memory\_dataset underflow due to miscalculated used\_memory\_overhead ([#​3005](valkey-io/valkey#3005)) - Avoid duplicate calculations of network-bytes-out in slot stats with copy-avoidance ([#​3046](valkey-io/valkey#3046)) - Fix XREAD returning error on empty stream with + ID ([#​2742](valkey-io/valkey#2742)) #### Performance/Efficiency Improvements - Track reply bytes in I/O threads if commandlog-reply-larger-than is -1 ([#​3086](valkey-io/valkey#3086), [#​3126](valkey-io/valkey#3126)). This makes it possible to mitigate a performance regression in 9.0.1 caused by the bug fix [#​2652](valkey-io/valkey#2652). **Full Changelog**: <valkey-io/valkey@9.0.1...9.0.2> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 6am" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xNjkuNCIsInVwZGF0ZWRJblZlciI6IjQzLjE2OS40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZSJdfQ==-->
In
dbSetValueif the robj is a hash with volatile fields we need to clean it up. It causes two core problems:keys_with_volatile_itemstrackedhashto ie. a string breaks this askeys_with_volatile_itemsis not cleaned up.Check #3000