Skip to content

Add support for multi-valued tables#89728

Merged
sokra merged 5 commits intocanaryfrom
multi_valued_families
Mar 3, 2026
Merged

Add support for multi-valued tables#89728
sokra merged 5 commits intocanaryfrom
multi_valued_families

Conversation

@lukesandberg
Copy link
Contributor

@lukesandberg lukesandberg commented Feb 9, 2026

What

Add support for multi-valued tables in turbo-persistence.

A multi-valued table allows multiple distinct values to be associated with a single key. Each family is independently configured as SingleValue (existing behavior) or MultiValue via the new FamilyKind enum.

Why

This will support the TaskCache table (implemented in #88904), where keys will change to be hashes instead of full TaskType values. This greatly decreases DB size and speeds up queries due to smaller key sizes, at the cost of hash collisions requiring multiple values per key.

How

API

  • New FamilyKind enum (SingleValue / MultiValue) and per-family FamilyConfig in DbConfig
  • get() for single-valued families (panics if called on multi-valued)
  • get_multiple() for multi-valued families, returns SmallVec<[ArcBytes; 1]> — stack-allocated for the common 0–1 result case, heap-scales when needed
  • put() and delete() are unchanged — the family kind controls dedup/compaction behavior

Write path & compaction

  • Single-valued (unchanged): last-write-wins per key
  • Multi-valued: all values are maintained, deletions 'shadow' old values.
  • Deletion inserts a tombstone that shadows all older values for that key across SST layers. Values written after the tombstone in the same batch are retained.
  • To avoid extra buffering logic the MergeIter semantics were changed so it produces 'newest' entries first
    • for SingleValues families this is no different since we only keep one value
    • for MultiValued families this makes dealing with tombstones trivial, but does mean compaction will reverse the order of the set. For this reason we make no guarantees about ordering.

Read path

  • Controlled by a FIND_ALL const generic on the internal lookup methods
  • Single-valued (FIND_ALL=false): binary search, return last match, stop
    • This fixes a bug where we might return Deleted when there is a value in the SST depending on what the search algorithm found first
  • Multi-valued (FIND_ALL=true): scan all matching entries in the SST block, then continue to older SSTs. If a tombstone is found, stop searching older layers.

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 9, 2026

Merging this PR will not alter performance

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks1


Comparing multi_valued_families (cfdb403) with canary (bbb72bf)

Open in CodSpeed

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@nextjs-bot
Copy link
Collaborator

nextjs-bot commented Feb 9, 2026

Stats from current PR

🔴 1 regression

Metric Canary PR Change Trend
Cold (First Request) 1.683s 1.803s 🔴 +120ms (+7%) ▂▁▂▁▁
📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 966ms 963ms ▁▁▁▁▁
Cold (Ready in log) 970ms 940ms ▁▁▁▁▁
Cold (First Request) 1.683s 1.803s 🔴 +120ms (+7%) ▂▁▂▁▁
Warm (Listen) 913ms 962ms ▁▁▁▁▁
Warm (Ready in log) 903ms 956ms ▁▁▁▁▁
Warm (First Request) 742ms 767ms ▁▁▁▁▁
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 456ms 455ms ▁▁▁▁█
Cold (Ready in log) 439ms 440ms ▁▄▄▁▆
Cold (First Request) 1.949s 1.945s ▁▂▃▁▇
Warm (Listen) 456ms 455ms ▁▁▁▁█
Warm (Ready in log) 440ms 440ms ▁▅▄▁▇
Warm (First Request) 1.941s 1.950s ▁▃▃▁▆

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 6.920s 7.117s ▁▁▁▁▁
Cached Build 6.801s 6.676s ▁▁▁▁▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 14.118s 14.134s ▁▁▃▁▆
Cached Build 14.184s 14.160s ▁▁▃▁▆
node_modules Size 476 MB 476 MB ▁▁▁▁▁
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **401 kB** → **401 kB** ✅ -23 B

80 files with content-based hashes (individual files not comparable between builds)

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 768 B 765 B
Total 768 B 765 B ✅ -3 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 450 B 450 B
Total 450 B 450 B

📦 Webpack

Client

Main Bundles
Canary PR Change
5528-HASH.js gzip 5.54 kB N/A -
6280-HASH.js gzip 58.7 kB N/A -
6335.HASH.js gzip 169 B N/A -
912-HASH.js gzip 4.59 kB N/A -
e8aec2e4-HASH.js gzip 62.6 kB N/A -
framework-HASH.js gzip 59.7 kB 59.7 kB
main-app-HASH.js gzip 255 B 253 B
main-HASH.js gzip 39.1 kB 39.1 kB
webpack-HASH.js gzip 1.68 kB 1.68 kB
262-HASH.js gzip N/A 4.59 kB -
2889.HASH.js gzip N/A 169 B -
5602-HASH.js gzip N/A 5.55 kB -
6948ada0-HASH.js gzip N/A 62.6 kB -
9544-HASH.js gzip N/A 59.4 kB -
Total 232 kB 233 kB ⚠️ +731 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB
Total 39.4 kB 39.4 kB
Pages
Canary PR Change
_app-HASH.js gzip 194 B 194 B
_error-HASH.js gzip 183 B 180 B 🟢 3 B (-2%)
css-HASH.js gzip 331 B 330 B
dynamic-HASH.js gzip 1.81 kB 1.81 kB
edge-ssr-HASH.js gzip 256 B 256 B
head-HASH.js gzip 351 B 352 B
hooks-HASH.js gzip 384 B 383 B
image-HASH.js gzip 580 B 581 B
index-HASH.js gzip 260 B 260 B
link-HASH.js gzip 2.5 kB 2.5 kB
routerDirect..HASH.js gzip 320 B 319 B
script-HASH.js gzip 386 B 386 B
withRouter-HASH.js gzip 315 B 315 B
1afbb74e6ecf..834.css gzip 106 B 106 B
Total 7.97 kB 7.97 kB ✅ -2 B

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 125 kB 125 kB
page.js gzip 255 kB 255 kB
Total 380 kB 380 kB ⚠️ +282 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 618 B 614 B
middleware-r..fest.js gzip 156 B 155 B
middleware.js gzip 43.8 kB 43.7 kB
edge-runtime..pack.js gzip 842 B 842 B
Total 45.4 kB 45.3 kB ✅ -113 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 715 B 718 B
Total 715 B 718 B ⚠️ +3 B
Build Cache
Canary PR Change
0.pack gzip 4.05 MB 4.06 MB 🔴 +6.29 kB (+0%)
index.pack gzip 103 kB 102 kB
index.pack.old gzip 103 kB 103 kB
Total 4.26 MB 4.26 MB ⚠️ +5.31 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 321 kB 321 kB
app-page-exp..prod.js gzip 170 kB 170 kB
app-page-tur...dev.js gzip 320 kB 320 kB
app-page-tur..prod.js gzip 170 kB 170 kB
app-page-tur...dev.js gzip 317 kB 317 kB
app-page-tur..prod.js gzip 168 kB 168 kB
app-page.run...dev.js gzip 317 kB 317 kB
app-page.run..prod.js gzip 168 kB 168 kB
app-route-ex...dev.js gzip 70.8 kB 70.8 kB
app-route-ex..prod.js gzip 49.3 kB 49.3 kB
app-route-tu...dev.js gzip 70.9 kB 70.9 kB
app-route-tu..prod.js gzip 49.3 kB 49.3 kB
app-route-tu...dev.js gzip 70.5 kB 70.5 kB
app-route-tu..prod.js gzip 49 kB 49 kB
app-route.ru...dev.js gzip 70.4 kB 70.4 kB
app-route.ru..prod.js gzip 49 kB 49 kB
dist_client_...dev.js gzip 324 B 324 B
dist_client_...dev.js gzip 326 B 326 B
dist_client_...dev.js gzip 318 B 318 B
dist_client_...dev.js gzip 317 B 317 B
pages-api-tu...dev.js gzip 43.2 kB 43.2 kB
pages-api-tu..prod.js gzip 32.9 kB 32.9 kB
pages-api.ru...dev.js gzip 43.2 kB 43.2 kB
pages-api.ru..prod.js gzip 32.9 kB 32.9 kB
pages-turbo....dev.js gzip 52.6 kB 52.6 kB
pages-turbo...prod.js gzip 38.5 kB 38.5 kB
pages.runtim...dev.js gzip 52.6 kB 52.6 kB
pages.runtim..prod.js gzip 38.5 kB 38.5 kB
server.runti..prod.js gzip 62 kB 62 kB
Total 2.83 MB 2.83 MB ⚠️ +7 B
📝 Changed Files (4 files)

Files with changes:

  • pages-api-tu..ntime.dev.js
  • pages-turbo...ntime.dev.js
  • pages-api.ru..time.prod.js
  • pages.runtime.prod.js
View diffs
pages-api-tu..ntime.dev.js

Diff too large to display

pages-turbo...ntime.dev.js

Diff too large to display

pages-api.ru..time.prod.js

Diff too large to display

pages.runtime.prod.js

Diff too large to display

📎 Tarball URL
https://vercel-packages.vercel.app/next/commits/cfdb4037ddd7a7d5744b3ff3c71a91b3c8cbc3fd/next

@lukesandberg lukesandberg changed the base branch from canary to graphite-base/89728 February 11, 2026 20:57
@lukesandberg lukesandberg changed the base branch from graphite-base/89728 to compaction-fixes February 11, 2026 20:57
@lukesandberg lukesandberg changed the base branch from compaction-fixes to graphite-base/89728 February 12, 2026 00:29
@lukesandberg lukesandberg force-pushed the multi_valued_families branch 2 times, most recently from 8c46c90 to 267e7d2 Compare March 2, 2026 03:23
@lukesandberg lukesandberg requested a review from sokra March 2, 2026 03:35
@lukesandberg lukesandberg force-pushed the multi_valued_families branch from 267e7d2 to 33486a5 Compare March 2, 2026 03:36
lukesandberg and others added 2 commits March 2, 2026 13:47
- Drop value deduplication: DB no longer deduplicates multi-values;
  that responsibility moves to callers
- Strict collector semantics: Delete only shadows pre-existing keys;
  duplicate inserts in SingleValue families trigger debug assertion
- Tombstone ordering: Collector places tombstones last in key groups
  for correct MergeIter compaction behavior
- Meta file tombstone handling: Early return when tombstone found in
  FIND_ALL mode to avoid searching older SSTs
- SST lookup update: Updated FIND_ALL scan for tombstone-last ordering
- Compaction split boundary: Don't split SSTs mid-key-group by only
  checking fullness at key boundaries
- Simplified get_impl tombstone handling by returning directly
- Updated tests to reflect new WriteBatch invariants
@lukesandberg lukesandberg force-pushed the multi_valued_families branch from 5740e92 to c944fe9 Compare March 2, 2026 21:48
lukesandberg and others added 3 commits March 2, 2026 13:53
A race condition in get_or_create_persistent_task() can cause the same
(task_type, task_id) pair to be pushed to persisted_task_cache_log twice
when two threads concurrently miss the task_cache fast path. This causes
a WriteBatch invariant violation (duplicate key in SingleValue family)
in the turbo-persistence collector.

Fix by tracking seen TaskIds with an FxHashSet in both the concurrent
and serial write paths of save_snapshot, skipping duplicates before
they reach WriteBatch::put().
Replace per-test-case family numbers with a unique prefix byte on family
0 to avoid key collisions when test cases share a single database. The
"Families" test case is unchanged since it specifically tests multi-family
support.
@sokra sokra merged commit 11823f8 into canary Mar 3, 2026
283 of 287 checks passed
@sokra sokra deleted the multi_valued_families branch March 3, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants