Fix used_memory_dataset underflow due to miscalculated used_memory_overhead by bpint · Pull Request #3005 · valkey-io/valkey

bpint · 2026-01-05T03:59:14Z

Describe the bug

The metric used_memory_dataset turned into an insanely large number close to 2^64 (actually overflowed negative value),
as reported in #2994 (comment).

Versions affected include 8.x and 9.x. (Should i submit multiple PRs?)

Analysis

We found that miscalculated used_memory_overhead is the core reason.

Double-Counted database memory

When server starts, the global variable server.initial_memory_usage is used to record a memory baseline in InitServerLast(). This server.initial_memory_usage has clearly included initial database memory, since databases are created in initServer().

In function getMemoryOverheadData(), the mem_total is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE.

This eventually caused wrongly larger used_memory_overhead. For a database with only a couple of keys, the used_memory_overhead is easily larger than used_memory and causes an overflowed used_memory_dataset.

Missed Empty Databases

In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation,

if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue;

However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including hashtable_size_index, which can be larger than 128 KiB.

On the contrary, this caused wrongly smaller used_memory_overhead for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and used_memory_overhead will increase (for used_memory_dataset decrease) by more than 128 KiB due to the single key insertion.

Proposed Fix

In this PR,

a) We record the zmalloc_used_memory() change before and after database creation during server initialization. And later when we decide the baseline server.initial_memory_usage, the initial database memory usage is excluded.

b) We remove the condition kvstoreNumAllocatedHashtables() in getMemoryOverheadData(), to make the used_memory_overhead reasonable for both empty and non-empty kvstores.

codecov · 2026-01-05T05:10:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.26%. Comparing base (e4a3e9f) to head (bc97c82).
⚠️ Report is 21 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #3005      +/-   ##
============================================
- Coverage     74.34%   74.26%   -0.09%     
============================================
  Files           129      129              
  Lines         70908    70914       +6     
============================================
- Hits          52714    52661      -53     
- Misses        18194    18253      +59

Files with missing lines	Coverage Δ
src/object.c	`91.52% <100.00%> (ø)`
src/server.c	`89.44% <100.00%> (+0.01%)`	⬆️

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

madolson · 2026-01-05T05:40:44Z

Versions affected include 8.x and 9.x. (Should i submit multiple PRs?)

We'll backport it as needed after the main PR is submitted.

…overhead Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

Corrected spelling of `necessary` to `necessary` in comments. Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

enjoy-binbin

LGTM, good catch.

Co-authored-by: Binbin <binloveplay1314@qq.com> Signed-off-by: bpint <chemistudio@gmail.com>

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

enjoy-binbin

Would you able to write a TCL test case to cover this? If not, i can take a look later.

bpint · 2026-01-06T04:19:15Z

Would you able to write a TCL test case to cover this? If not, i can take a look later.

really sorry that i am familiar with neither tcl language nor the principle of test cases.

enjoy-binbin · 2026-01-15T07:48:16Z

@bpint Merged, thank you.

bpint · 2026-01-15T10:24:43Z

respect to @madolson @enjoy-binbin and all maintainers for your efficient and awesome work

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: arshidkv12 <arshidkv12@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in #2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…erhead (#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in #2994. When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…erhead (valkey-io#3005) The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually overflowed negative value), as reported in valkey-io#2994. ## Double-Counted database memory When server starts, the global variable `server.initial_memory_usage` is used to record a memory baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database memory, since databases are created in initServer. In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes initial database memory. And then all extra memory usage of databases are added to mem_total. The initial database memory are therefore counted TWICE. This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys, the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`. ## Missed Empty Databases In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation: ```c if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue; ``` However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(), including `hashtable_size_index`, which can be larger than 128 KiB. On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase (for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion. Signed-off-by: Ace Breakpoint <chemistudio@gmail.com> Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Binbin <binloveplay1314@qq.com> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [docker.io/valkey/valkey](https://github.com/valkey-io/valkey) | image | patch | `9.0.1` → `9.0.4` | --- ### Release Notes <details> <summary>valkey-io/valkey (docker.io/valkey/valkey)</summary> ### [`v9.0.4`](https://github.com/valkey-io/valkey/releases/tag/9.0.4) [Compare Source](valkey-io/valkey@9.0.3...9.0.4) Upgrade urgency SECURITY: This release includes security fixes we recommend you apply as soon as possible. ##### Security fixes - (CVE-2026-23479) Use-After-Free in unblock client flow - (CVE-2026-25243) Invalid Memory Access in RESTORE command - (CVE-2026-23631) Use-after-free when full sync occurs during a yielding Lua/function execution ### [`v9.0.3`](https://github.com/valkey-io/valkey/releases/tag/9.0.3) [Compare Source](valkey-io/valkey@9.0.2...9.0.3) ##### Valkey 9.0.3 Upgrade urgency SECURITY: This release includes security fixes we recommend you apply as soon as possible. ##### Security fixes - (CVE-2025-67733) RESP Protocol Injection via Lua error\_reply - (CVE-2026-21863) Remote DoS with malformed Valkey Cluster bus message - (CVE-2026-27623) Reset request type after handling empty requests ##### Bug fixes - Avoids crash during MODULE UNLOAD when ACL rules reference a module command and subcommand ([#3160](valkey-io/valkey#3160)) - Fix server assert on ACL LOAD when current user loses permission to channels ([#3182](valkey-io/valkey#3182)) - Fix bug causing no response flush sometimes when IO threads are busy ([#3205](valkey-io/valkey#3205)) ### [`v9.0.2`](https://github.com/valkey-io/valkey/releases/tag/9.0.2) [Compare Source](valkey-io/valkey@9.0.1...9.0.2) Upgrade urgency HIGH: There are critical bugs that may affect a subset of users. #### Bug fixes - Avoid memory leak of new argv when HEXPIRE commands target only non-exiting fields ([#2973](valkey-io/valkey#2973)) - Fix HINCRBY and HINCRBYFLOAT to update volatile key tracking ([#2974](valkey-io/valkey#2974)) - Avoid empty hash object when HSETEX added no fields ([#2998](valkey-io/valkey#2998)) - Fix case-sensitive check for the FNX and FXX arguments in HSETEX ([#3000](valkey-io/valkey#3000)) - Prevent assertion in active expiration job after a hash with volatile fields is overwritten ([#3003](valkey-io/valkey#3003), [#3007](valkey-io/valkey#3007)) - Fix HRANDFIELD to return null response when no field could be found ([#3022](valkey-io/valkey#3022)) - Fix HEXPIRE to not delete items when validation rules fail and expiration is in the past ([#3023](valkey-io/valkey#3023), [#3048](valkey-io/valkey#3048)) - Fix how hash is handling overriding of expired fields overwrite ([#3060](valkey-io/valkey#3060)) - HSETEX - Always issue keyspace notifications after validation ([#3001](valkey-io/valkey#3001)) - Make zero a valid TTL for hash fields during import mode and data loading ([#3006](valkey-io/valkey#3006)) - Trigger prepareCommand on argc change in module command filters ([#2945](valkey-io/valkey#2945)) - Restrict TTL from being negative and avoid crash in import-mode ([#2944](valkey-io/valkey#2944)) - Fix chained replica crash when doing dual channel replication ([#2983](valkey-io/valkey#2983)) - Skip slot cache optimization for AOF client to prevent key duplication and data corruption ([#3004](valkey-io/valkey#3004)) - Fix used\_memory\_dataset underflow due to miscalculated used\_memory\_overhead ([#3005](valkey-io/valkey#3005)) - Avoid duplicate calculations of network-bytes-out in slot stats with copy-avoidance ([#3046](valkey-io/valkey#3046)) - Fix XREAD returning error on empty stream with + ID ([#2742](valkey-io/valkey#2742)) #### Performance/Efficiency Improvements - Track reply bytes in I/O threads if commandlog-reply-larger-than is -1 ([#3086](valkey-io/valkey#3086), [#3126](valkey-io/valkey#3126)). This makes it possible to mitigate a performance regression in 9.0.1 caused by the bug fix [#2652](valkey-io/valkey#2652). **Full Changelog**: <valkey-io/valkey@9.0.1...9.0.2> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 6am" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).

bpint mentioned this pull request Jan 5, 2026

[BUG] Serious and Curious Memory Statistic Problem of used_memory and used_memory_dataset #2994

Closed

github-actions Bot assigned bpint Jan 5, 2026

bpint force-pushed the unstable branch from 9d82c3d to 0887393 Compare January 5, 2026 05:19

madolson reviewed Jan 5, 2026

View reviewed changes

Comment thread src/server.c Outdated

bpint added 3 commits January 5, 2026 15:20

Fix underflowed used_memory_dataset due to miscalculated used_memory_…

7c3eddb

…overhead Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

Fix typo in initial_memory_usage comment

f167f3e

Corrected spelling of `necessary` to `necessary` in comments. Signed-off-by: bpint <chemistudio@gmail.com> Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

better initialization of initial_memory_usage as @madolson suggested

fd5ae7e

Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>

bpint force-pushed the unstable branch from 9164658 to fd5ae7e Compare January 5, 2026 07:21

bpint requested a review from madolson January 5, 2026 09:54

enjoy-binbin approved these changes Jan 5, 2026

View reviewed changes

Comment thread src/server.c Outdated

Comment thread src/object.c

Comment thread src/object.c Outdated

bpint and others added 2 commits January 5, 2026 18:12

clarify comments as suggested by @enjoy-binbin

f624801

Co-authored-by: Binbin <binloveplay1314@qq.com> Signed-off-by: bpint <chemistudio@gmail.com>

Remove comments as suggested by @enjoy-binbin

bc85cc0

Co-authored-by: Binbin <binloveplay1314@qq.com> Signed-off-by: bpint <chemistudio@gmail.com>

bpint requested a review from enjoy-binbin January 5, 2026 10:17

enjoy-binbin added this to Valkey 8.1, Valkey 9.0 and Valkey 8.0 Jan 5, 2026

enjoy-binbin moved this to To be backported in Valkey 8.0 Jan 5, 2026

enjoy-binbin moved this to To be backported in Valkey 8.1 Jan 5, 2026

enjoy-binbin moved this to To be backported in Valkey 9.0 Jan 5, 2026

enjoy-binbin added the release-notes This issue should get a line item in the release notes label Jan 5, 2026

madolson reviewed Jan 5, 2026

View reviewed changes

Comment thread src/server.c Outdated

Update src/server.c

bc97c82

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>

enjoy-binbin approved these changes Jan 6, 2026

View reviewed changes

enjoy-binbin merged commit 2635377 into valkey-io:unstable Jan 15, 2026
24 checks passed

zuiderkwast linked an issue Jan 27, 2026 that may be closed by this pull request

[BUG] Serious and Curious Memory Statistic Problem of used_memory and used_memory_dataset #2994

Closed

roshkhatri moved this from To be backported to 8.1.6 in Valkey 8.1 Jan 29, 2026

roshkhatri moved this from To be backported to 8.0.7 in Valkey 8.0 Jan 30, 2026

zuiderkwast moved this from To be backported to 9.0.2 WIP in Valkey 9.0 Jan 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix used_memory_dataset underflow due to miscalculated used_memory_overhead#3005

Fix used_memory_dataset underflow due to miscalculated used_memory_overhead#3005
enjoy-binbin merged 6 commits into
valkey-io:unstablefrom
bpint:unstable

bpint commented Jan 5, 2026

Uh oh!

codecov Bot commented Jan 5, 2026 •

edited

Loading

Uh oh!

madolson commented Jan 5, 2026

Uh oh!

Uh oh!

enjoy-binbin left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

enjoy-binbin left a comment

Uh oh!

bpint commented Jan 6, 2026

Uh oh!

Uh oh!

enjoy-binbin commented Jan 15, 2026

Uh oh!

bpint commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

bpint commented Jan 5, 2026

Describe the bug

Analysis

Double-Counted database memory

Missed Empty Databases

Proposed Fix

Uh oh!

codecov Bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

madolson commented Jan 5, 2026

Uh oh!

Uh oh!

enjoy-binbin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

enjoy-binbin left a comment

Choose a reason for hiding this comment

Uh oh!

bpint commented Jan 6, 2026

Uh oh!

Uh oh!

enjoy-binbin commented Jan 15, 2026

Uh oh!

bpint commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Jan 5, 2026 •

edited

Loading