Skip to content

Create a per family configuration that will allow us to optimize the file sizes for our different file types.#89058

Closed
lukesandberg wants to merge 2 commits intographite-base/89058from
family_size_configuration
Closed

Create a per family configuration that will allow us to optimize the file sizes for our different file types.#89058
lukesandberg wants to merge 2 commits intographite-base/89058from
family_size_configuration

Conversation

@lukesandberg
Copy link
Contributor

@lukesandberg lukesandberg commented Jan 26, 2026 •
edited
Loading

Add configurable file limits to TurboPersistence

What?

Adds per-family configuration options to TurboPersistence to control when SST files are split during writes. Use this to decrease the number of SST files for TaskCache

Why?

Different keyspaces have different access patterns and data characteristics. This change allows optimizing file size limits for each family to improve performance and memory usage.

@nextjs-bot nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Jan 26, 2026
@lukesandberg lukesandberg mentioned this pull request Jan 26, 2026
Merged
@lukesandberg Graphite App
Copy link
Contributor Author

lukesandberg commented Jan 26, 2026 •
edited
Loading

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Jan 26, 2026
Merged
Merged
@codspeed-hq
Copy link

codspeed-hq bot commented Jan 26, 2026 •
edited
Loading

CodSpeed Performance Report

Merging this PR will not alter performance

Comparing family_size_configuration (49f013c) with canary (7c9be45)1

Summary

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks2

Footnotes

  1. No successful run was found on graphite-base/89058 (439f2ab) during the generation of this report, so canary (7c9be45) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

  2. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@nextjs-bot
Copy link
Collaborator

nextjs-bot commented Jan 26, 2026 •
edited
Loading

Stats from current PR

✅ No significant changes detected

📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 455ms 456ms ✓ ▁▁▁▁▁
Cold (Ready in log) 439ms 438ms ✓ ▁▂▂▁▁
Cold (First Request) 1.165s 1.153s ✓ ▁▄▃▃▃
Warm (Listen) 457ms 457ms ✓ ▁▁▁▁▁
Warm (Ready in log) 443ms 443ms ✓ ▁▁▁▁▁
Warm (First Request) 340ms 340ms ✓ ▁▁▁▁▁
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 455ms 456ms ✓ ▁▁▅▁▁
Cold (Ready in log) 439ms 439ms ✓ ▃▃▅▂▄
Cold (First Request) 1.886s 1.907s ✓ ▂▂▆▂▂
Warm (Listen) 455ms 455ms ✓ ▁▁▄▁▁
Warm (Ready in log) 438ms 438ms ✓ ▃▃▅▂▄
Warm (First Request) 1.901s 1.875s ✓ ▂▂▅▁▂

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 3.818s 3.823s ✓ ▁▁▁▁▁
Cached Build 3.814s 3.854s ✓ ▁▁▁▁▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 13.876s 13.846s ✓ ▁▁▅▁▁
Cached Build 14.003s 13.975s ✓ ▁▁▅▁▁
node_modules Size 467 MB 467 MB ✓ ▁▁▁▁▁
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **437 kB** → **437 kB** ✅ -1 B

81 files with content-based hashes (individual files not comparable between builds)

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 765 B 767 B ✓
Total 765 B 767 B ⚠️ +2 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 451 B 450 B ✓
Total 451 B 450 B ✅ -1 B

📦 Webpack

Client

Main Bundles
Canary PR Change
5528-HASH.js gzip 5.47 kB N/A -
6280-HASH.js gzip 56.9 kB N/A -
6335.HASH.js gzip 169 B N/A -
912-HASH.js gzip 4.53 kB N/A -
e8aec2e4-HASH.js gzip 62.5 kB N/A -
framework-HASH.js gzip 59.7 kB 59.7 kB ✓
main-app-HASH.js gzip 256 B 253 B 🟢 3 B (-1%)
main-HASH.js gzip 39.1 kB 39.1 kB ✓
webpack-HASH.js gzip 1.68 kB 1.68 kB ✓
262-HASH.js gzip N/A 4.52 kB -
2889.HASH.js gzip N/A 169 B -
5602-HASH.js gzip N/A 5.48 kB -
6948ada0-HASH.js gzip N/A 62.5 kB -
9544-HASH.js gzip N/A 57.5 kB -
Total 230 kB 231 kB ⚠️ +614 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB ✓
Total 39.4 kB 39.4 kB ✓
Pages
Canary PR Change
_app-HASH.js gzip 194 B 194 B ✓
_error-HASH.js gzip 183 B 180 B 🟢 3 B (-2%)
css-HASH.js gzip 331 B 330 B ✓
dynamic-HASH.js gzip 1.81 kB 1.81 kB ✓
edge-ssr-HASH.js gzip 256 B 256 B ✓
head-HASH.js gzip 351 B 352 B ✓
hooks-HASH.js gzip 384 B 383 B ✓
image-HASH.js gzip 580 B 581 B ✓
index-HASH.js gzip 260 B 260 B ✓
link-HASH.js gzip 2.49 kB 2.49 kB ✓
routerDirect..HASH.js gzip 320 B 319 B ✓
script-HASH.js gzip 386 B 386 B ✓
withRouter-HASH.js gzip 315 B 315 B ✓
1afbb74e6ecf..834.css gzip 106 B 106 B ✓
Total 7.97 kB 7.97 kB ✅ -1 B

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 126 kB 126 kB ✓
page.js gzip 249 kB 249 kB ✓
Total 375 kB 375 kB ⚠️ +476 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 615 B 616 B ✓
middleware-r..fest.js gzip 156 B 155 B ✓
middleware.js gzip 33.1 kB 33.3 kB ✓
edge-runtime..pack.js gzip 842 B 842 B ✓
Total 34.7 kB 34.9 kB ⚠️ +196 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 732 B 736 B ✓
Total 732 B 736 B ⚠️ +4 B
Build Cache
Canary PR Change
0.pack gzip 3.84 MB 3.85 MB 🔴 +8.31 kB (+0%)
index.pack gzip 102 kB 103 kB ✓
index.pack.old gzip 103 kB 101 kB 🟢 1.31 kB (-1%)
Total 4.05 MB 4.05 MB ⚠️ +7.07 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 315 kB 315 kB ✓
app-page-exp..prod.js gzip 167 kB 167 kB ✓
app-page-tur...dev.js gzip 315 kB 315 kB ✓
app-page-tur..prod.js gzip 167 kB 167 kB ✓
app-page-tur...dev.js gzip 312 kB 312 kB ✓
app-page-tur..prod.js gzip 165 kB 165 kB ✓
app-page.run...dev.js gzip 312 kB 312 kB ✓
app-page.run..prod.js gzip 165 kB 165 kB ✓
app-route-ex...dev.js gzip 70.5 kB 70.5 kB ✓
app-route-ex..prod.js gzip 49 kB 49 kB ✓
app-route-tu...dev.js gzip 70.5 kB 70.5 kB ✓
app-route-tu..prod.js gzip 49 kB 49 kB ✓
app-route-tu...dev.js gzip 70.1 kB 70.1 kB ✓
app-route-tu..prod.js gzip 48.8 kB 48.8 kB ✓
app-route.ru...dev.js gzip 70.1 kB 70.1 kB ✓
app-route.ru..prod.js gzip 48.7 kB 48.7 kB ✓
dist_client_...dev.js gzip 324 B 324 B ✓
dist_client_...dev.js gzip 326 B 326 B ✓
dist_client_...dev.js gzip 318 B 318 B ✓
dist_client_...dev.js gzip 317 B 317 B ✓
pages-api-tu...dev.js gzip 43.2 kB 43.2 kB ✓
pages-api-tu..prod.js gzip 32.9 kB 32.9 kB ✓
pages-api.ru...dev.js gzip 43.1 kB 43.1 kB ✓
pages-api.ru..prod.js gzip 32.8 kB 32.8 kB ✓
pages-turbo....dev.js gzip 52.5 kB 52.5 kB ✓
pages-turbo...prod.js gzip 39.4 kB 39.4 kB ✓
pages.runtim...dev.js gzip 52.5 kB 52.5 kB ✓
pages.runtim..prod.js gzip 39.3 kB 39.3 kB ✓
server.runti..prod.js gzip 62.7 kB 62.7 kB ✓
Total 2.8 MB 2.8 MB ✅ -2 B

@lukesandberg lukesandberg force-pushed the family_size_configuration branch from 1750d5e to 83303ff Compare January 26, 2026 19:07
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from 83303ff to a974ae3 Compare January 26, 2026 21:49
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from a974ae3 to 9dfbd35 Compare January 26, 2026 23:27
@lukesandberg lukesandberg mentioned this pull request Jan 27, 2026
Closed
@lukesandberg lukesandberg changed the base branch from remove_backend_field to graphite-base/89058 January 27, 2026 00:39
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from 9dfbd35 to 9ae341f Compare January 27, 2026 00:39
@lukesandberg lukesandberg changed the base branch from graphite-base/89058 to persistent_hash January 27, 2026 00:40
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from 9ae341f to fd272b9 Compare January 27, 2026 00:48
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from fd272b9 to 4daa326 Compare January 27, 2026 00:59
@lukesandberg lukesandberg force-pushed the family_size_configuration branch 2 times, most recently from ec1dad5 to f502c31 Compare January 27, 2026 01:06
@lukesandberg lukesandberg force-pushed the persistent_hash branch 2 times, most recently from 6cade98 to 9fe6b3c Compare January 29, 2026 19:32
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from f502c31 to cc8a6aa Compare January 29, 2026 19:32
@lukesandberg lukesandberg mentioned this pull request Jan 29, 2026
Merged
@lukesandberg lukesandberg force-pushed the family_size_configuration branch from ebe3e36 to d482259 Compare January 30, 2026 01:26
This was referenced Jan 30, 2026
Merged
Merged
sokra
sokra requested changes Feb 3, 2026
View reviewed changes
Copy link
Member

@sokra sokra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's weird to need different heuristic per family.

I think we should have heuristics that work in general and doesn't require fine-tuning.
Maybe we need a more adaptive heuristic?
We already have max entries and max data. With these we should be able to construct a heuristic that is adaptive to small vs big entries.
We might need to be more adaptive to small entry amount (e.g. infra, but this can also happen for all families as incremental builds might write only a few entries). Maybe we need to buffer to start small and grow with more entries.

@lukesandberg lukesandberg force-pushed the family_size_configuration branch from d482259 to 49f013c Compare February 8, 2026 19:17
@lukesandberg lukesandberg changed the base branch from persistent_hash to graphite-base/89058 February 8, 2026 19:24
@lukesandberg lukesandberg mentioned this pull request Feb 9, 2026
Merged
@lukesandberg lukesandberg mentioned this pull request Feb 11, 2026
Closed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 26, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@sokra sokra sokra requested changes

Assignees

No one assigned

Labels

created-by: Turbopack team PRs by the Turbopack team. locked Turbopack Related to Turbopack with Next.js.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lukesandberg @nextjs-bot @sokra