feat: add TaskStorage derive macro and schema infrastructure #88338

lukesandberg · 2026-01-09T17:36:59Z

Add a new derive macro for TaskStorage a replacement for InnerStorage and CachedDataItem, currently it is unused outside of tests.

What

add a new derive macro that generates code from the 'schema' in storage_schema.rs
storage_schema.rs defines the layout of our new struct, for each field we specify
- type and name
- category ('data' 'meta' or 'transient')
- location (inline or lazy or flags for bitfields)
- whether we should filter_transient when encoding (not everything can contain transient data)
- Logically this is a replacement for CachedDataItem
From that we generate a few things
- TaskFlags a bitfield to hold some data items as well as 'meta' information like the 'snapshot' and 'restore' bits replacing InnerStorageState
- LazyField enum, defines tall the fields that we dynamically allocate
- TaskStorage the main storage struct, along with basic accessors for all the datatypes (which helps encapsualte storage strategy)
- TaskStorageAccessors a trait with default implementations of every accessor which also handle
  - access checking, via check_access
  - mutation tracking via track_mutation
  - putting these behaviors in a trait allows some of the business logic around snapshotting to remain in normal rust code that implements the trait, while all the field updating logic can be autogenerated
- Serialization and encoding infrastructure
  - routines to encode and decode the 'meta' or 'data' subsets
  - routines to create partial 'snapshots' of meta or data subsets
  - routines to restore data from deserialized snapshots

See https://gist.github.com/lukesandberg/ea35f0e92c1cdeddf80d0498f2978144 for a fully expanded version of the macro

Why

There are a few goals from this redesign of InnerStorage

(speculative) a single large derive macro will be easier to maintain than the current mix of macro_rules and derive macros we use for CacheDataItem
- There is one macro to expand when inspecting and the macro generation code is pretty straightforward if voluminous
We eliminate many match operations on CachedDataItem
- Today to lookup a piece of data we need to execute several matches to get the the data category and find the storage location. With this new approach those are handled by just calling the correct accessor which statically addresses the storage location
serialization performance
- we don't need to 'explode' the storage into Vec<CachedDataItem> anymore and instead can either directly snapshot the storage or encode directly to a buffer. This should eliminate some temporary allocations and should enable a smaller serialization format
more flexibility
- mostly this is the same approach as before with a few small improvements, notably invalidator and immutable are now stored as single inline bits reducing lookupcosts and memory for tasks that have these.

How

Most of the code was written by Claude with obviously a ton of feedback, so please review carefully. For now all this code is unused but future PRs will integrate it into the backend

nextjs-bot · 2026-01-09T17:37:25Z

Allow CI Workflow Run

approve CI run for commit: caab350

Note: this should only be enabled once the PR is ready to go and can only be enabled by a maintainer

nextjs-bot · 2026-01-09T17:37:25Z

Allow CI Workflow Run

approve CI run for commit: caab350

Note: this should only be enabled once the PR is ready to go and can only be enabled by a maintainer

lukesandberg · 2026-01-09T17:37:27Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs

nextjs-bot · 2026-01-09T17:47:27Z

Tests Passed

nextjs-bot · 2026-01-09T18:01:03Z

Stats from current PR

✅ No significant changes detected

📊 All Metrics

📖 Metrics Glossary

Dev Server Metrics:

Listen = TCP port starts accepting connections
First Request = HTTP server returns successful response
Cold = Fresh build (no cache)
Warm = With cached build artifacts

Build Metrics:

Fresh = Clean build (no .next directory)
Cached = With existing .next directory

Change Thresholds:

Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
Size: Changes < 1KB AND < 1% are insignificant
All other changes are flagged to catch regressions

⚡ Dev Server

Metric	Canary	PR	Change	Trend
Cold (Listen)	457ms	456ms	✓	▁█▁▁▁
Cold (Ready in log)	455ms	454ms	✓	▆▁▇█▆
Cold (First Request)	875ms	886ms	✓	█▁███
Warm (Listen)	456ms	457ms	✓	█▁▁█▁
Warm (Ready in log)	456ms	459ms	✓	▁█▂▄▃
Warm (First Request)	368ms	368ms	✓	▁█▄▇▁

📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric	Canary	PR	Change	Trend
Cold (Listen)	455ms	456ms	✓	▁▁▁▁▁
Cold (Ready in log)	451ms	452ms	✓	▄▃▂▁▁
Cold (First Request)	1.854s	1.848s	✓	▂▂▁▁▁
Warm (Listen)	456ms	455ms	✓	▁▁▁▁▁
Warm (Ready in log)	452ms	447ms	✓	▃▃▁▁▁
Warm (First Request)	1.843s	1.856s	✓	▂▂▁▁▁

⚡ Production Builds

Metric	Canary	PR	Change	Trend
Fresh Build	4.398s	4.359s	✓	▂█▁▂▂
Cached Build	4.342s	4.361s	✓	▂█▁▂▁

📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric	Canary	PR	Change	Trend
Fresh Build	14.183s	14.186s	✓	▁▁▁▁▁
Cached Build	14.302s	14.215s	✓	▁▁▁▁▁
node_modules Size	458 MB	458 MB	✓	▁▁▁▁▁

📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **430 kB** → **430 kB** ✅ -86 B

82 files with content-based hashes (individual files not comparable between builds)

Server

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	788 B	787 B	✓
Total	788 B	787 B	✅ -1 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	451 B	450 B	✓
Total	451 B	450 B	✅ -1 B

📦 Webpack

Client

Main Bundles

	Canary	PR	Change
2086.HASH.js gzip	169 B	N/A	-
2161-HASH.js gzip	5.41 kB	N/A	-
2747-HASH.js gzip	4.48 kB	N/A	-
4322-HASH.js gzip	52.3 kB	N/A	-
ec793fe8-HASH.js gzip	62.3 kB	N/A	-
framework-HASH.js gzip	59.8 kB	59.8 kB	✓
main-app-HASH.js gzip	251 B	254 B	🔴 +3 B (+1%)
main-HASH.js gzip	38.6 kB	38.9 kB	✓
webpack-HASH.js gzip	1.68 kB	1.68 kB	✓
1596.HASH.js gzip	N/A	169 B	-
2658-HASH.js gzip	N/A	51.9 kB	-
6349-HASH.js gzip	N/A	4.46 kB	-
7019-HASH.js gzip	N/A	5.43 kB	-
b17a3386-HASH.js gzip	N/A	62.3 kB	-
Total	225 kB	225 kB	✅ -11 B

Polyfills

	Canary	PR	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Total	39.4 kB	39.4 kB	✓

Pages

	Canary	PR	Change
_app-HASH.js gzip	194 B	193 B	✓
_error-HASH.js gzip	182 B	182 B	✓
css-HASH.js gzip	336 B	335 B	✓
dynamic-HASH.js gzip	1.8 kB	1.8 kB	✓
edge-ssr-HASH.js gzip	256 B	256 B	✓
head-HASH.js gzip	352 B	349 B	✓
hooks-HASH.js gzip	385 B	384 B	✓
image-HASH.js gzip	580 B	580 B	✓
index-HASH.js gzip	259 B	258 B	✓
link-HASH.js gzip	2.5 kB	2.51 kB	✓
routerDirect..HASH.js gzip	319 B	317 B	✓
script-HASH.js gzip	385 B	387 B	✓
withRouter-HASH.js gzip	316 B	315 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Total	7.97 kB	7.96 kB	✅ -8 B

Server

Edge SSR

	Canary	PR	Change
edge-ssr.js gzip	125 kB	125 kB	✓
page.js gzip	242 kB	237 kB	🟢 4.87 kB (-2%)
Total	366 kB	361 kB	✅ -4.85 kB

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	655 B	652 B	✓
middleware-r..fest.js gzip	155 B	156 B	✓
middleware.js gzip	33 kB	33.3 kB	✓
edge-runtime..pack.js gzip	842 B	842 B	✓
Total	34.7 kB	34.9 kB	⚠️ +275 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	738 B	738 B	✓
Total	738 B	738 B	✓

Build Cache

	Canary	PR	Change
0.pack gzip	3.66 MB	3.67 MB	🔴 +8.51 kB (+0%)
index.pack gzip	99.7 kB	99.9 kB	✓
index.pack.old gzip	99.1 kB	100 kB	🔴 +1.27 kB (+1%)
Total	3.86 MB	3.87 MB	⚠️ +9.99 kB

🔄 Shared (bundler-independent)

Runtimes

	Canary	PR	Change
app-page-exp...dev.js gzip	303 kB	303 kB	✓
app-page-exp..prod.js gzip	158 kB	158 kB	✓
app-page-tur...dev.js gzip	303 kB	303 kB	✓
app-page-tur..prod.js gzip	158 kB	158 kB	✓
app-page-tur...dev.js gzip	300 kB	300 kB	✓
app-page-tur..prod.js gzip	156 kB	156 kB	✓
app-page.run...dev.js gzip	300 kB	300 kB	✓
app-page.run..prod.js gzip	156 kB	156 kB	✓
app-route-ex...dev.js gzip	68.8 kB	68.8 kB	✓
app-route-ex..prod.js gzip	47.6 kB	47.6 kB	✓
app-route-tu...dev.js gzip	68.8 kB	68.8 kB	✓
app-route-tu..prod.js gzip	47.6 kB	47.6 kB	✓
app-route-tu...dev.js gzip	68.4 kB	68.4 kB	✓
app-route-tu..prod.js gzip	47.4 kB	47.4 kB	✓
app-route.ru...dev.js gzip	68.4 kB	68.4 kB	✓
app-route.ru..prod.js gzip	47.3 kB	47.3 kB	✓
dist_client_...dev.js gzip	324 B	324 B	✓
dist_client_...dev.js gzip	326 B	326 B	✓
dist_client_...dev.js gzip	318 B	318 B	✓
dist_client_...dev.js gzip	317 B	317 B	✓
pages-api-tu...dev.js gzip	41.2 kB	41.2 kB	✓
pages-api-tu..prod.js gzip	31.3 kB	31.3 kB	✓
pages-api.ru...dev.js gzip	41.1 kB	41.1 kB	✓
pages-api.ru..prod.js gzip	31.2 kB	31.2 kB	✓
pages-turbo....dev.js gzip	50.8 kB	50.8 kB	✓
pages-turbo...prod.js gzip	38.2 kB	38.2 kB	✓
pages.runtim...dev.js gzip	50.7 kB	50.7 kB	✓
pages.runtim..prod.js gzip	38.2 kB	38.2 kB	✓
server.runti..prod.js gzip	62.2 kB	62.2 kB	✓
Total	2.69 MB	2.69 MB	⚠️ +3 B

codspeed-hq · 2026-01-09T18:08:10Z

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing 01-09-task-storage-macro-infrastructure (54be541) with canary (b186a86)}

Summary

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks¹

3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs

bgw · 2026-01-13T21:41:45Z

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs

+        if self.is_transient() {
+            quote! { self.check_access(crate::backend::TaskDataCategory::All); }
+        } else if self.category == Category::Meta {
+            quote! { self.check_access(crate::backend::TaskDataCategory::Meta); }


Nit: Ideally things like crate::backend::TaskDataCategory would be a syn::Path or TokenStream passed into the macro, in case we wanted to use this macro outside of turbo-tasks-backend (e.g. for testing the macro in isolation).

yeah this is definitely an odd part of this, we are super tightly coupled. but im not too worried. i don't really foresee wanting to directly unit test this, but then again, i have been wondering if we should specialize for different task shapes.

all the same i think i will punt on this, if we come up with that second usecase we can add an option attribute (we would also need to move or export these particular types)

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs

Add the TaskStorage derive macro and storage schema as unused code scaffolding for the upcoming refactoring of InnerStorage. Changes: - Add turbo-tasks-macros/src/derive/task_storage_macro.rs (~2900 lines) - Supports inline/lazy storage modes - Generates typed accessors (get/set/take/ref/mut) - Generates encode_meta/encode_data/decode_meta/decode_data methods - Supports filter_transient attribute for skipping transient data - New `default` attribute for fields using Default::default() semantics - Add storage_schema.rs defining TaskStorage struct and schema - Defines all fields with storage attributes - Uses new `default` attribute for aggregation_number (12 bytes vs 16 with Option) - TaskStorage is 128 bytes (matches InnerStorage) - Includes comprehensive tests for accessors, encoding, and sizes - Add is_transient() methods to CellRef, CollectibleRef, CollectiblesRef, OutputValue - These are needed by the macro's filter_transient functionality 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

lukesandberg

thanks for all the feedfback, definitely an improvement and the questions allowed me to find some more simplifications in the macro

lukesandberg · 2026-01-15T03:47:38Z

Merge activity

Jan 15, 3:47 AM UTC: @lukesandberg merged this pull request with Graphite.

nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Jan 9, 2026

This was referenced Jan 9, 2026

fix(turbo-tasks-backend): use correct TaskDataCategory for is_immutable check #88331

Merged

perf(turbo-tasks-backend): use DefaultStorage for AggregationNumber to save memory #88336

Merged

lukesandberg mentioned this pull request Jan 9, 2026

docs: add rebuild guidance and ICE troubleshooting to AGENTS.md #88337

Merged

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from caab350 to 5398e9b Compare January 9, 2026 17:42

lukesandberg force-pushed the 01-09-agents-md-updates branch from a98f5fa to 00722c2 Compare January 9, 2026 17:42

vercel bot reviewed Jan 9, 2026

View reviewed changes

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs Outdated Show resolved Hide resolved

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from 102e146 to f86b6c5 Compare January 10, 2026 00:08

lukesandberg force-pushed the 01-09-agents-md-updates branch from 1c882f9 to 47d98d1 Compare January 10, 2026 00:08

lukesandberg mentioned this pull request Jan 10, 2026

feat: Replace InnerStorage with the generated TaskStorage struct #88355

Draft

lukesandberg marked this pull request as ready for review January 12, 2026 01:00

lukesandberg mentioned this pull request Jan 12, 2026

refactor: migrate to typed accessors and remove CachedDataItem adapter #88397

Draft

lukesandberg marked this pull request as draft January 12, 2026 05:51

lukesandberg marked this pull request as ready for review January 12, 2026 05:56

lukesandberg mentioned this pull request Jan 12, 2026

perf: optimize cell_data serialization format #88401

Draft

lukesandberg marked this pull request as draft January 12, 2026 07:31

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from f86b6c5 to dc32042 Compare January 12, 2026 07:43

lukesandberg marked this pull request as ready for review January 12, 2026 07:49

lukesandberg force-pushed the 01-09-agents-md-updates branch from 47d98d1 to c82b781 Compare January 12, 2026 16:57

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from dc32042 to 0d769e4 Compare January 12, 2026 16:57

lukesandberg force-pushed the 01-09-agents-md-updates branch 2 times, most recently from 26fb1db to 9300306 Compare January 12, 2026 17:48

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from 0d769e4 to f465836 Compare January 12, 2026 17:48

lukesandberg changed the base branch from 01-09-agents-md-updates to graphite-base/88338 January 12, 2026 18:14

lukesandberg force-pushed the graphite-base/88338 branch from 9300306 to d067b62 Compare January 12, 2026 18:15

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from f465836 to 20e2a0d Compare January 12, 2026 18:15

graphite-app bot changed the base branch from graphite-base/88338 to canary January 12, 2026 18:16

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from 20e2a0d to ef39cbd Compare January 12, 2026 18:16

lukesandberg requested a review from a team January 12, 2026 23:00

bgw reviewed Jan 13, 2026

View reviewed changes

bgw approved these changes Jan 13, 2026

View reviewed changes

lukesandberg and others added 8 commits January 14, 2026 13:13

simplify some of the macro configuration

9563a62

self review fixes

52a9642

typo

1bb10eb

improve validation

275eaac

a bit nicer to expect literal strings

ffadb9d

review feedback

f5467da

move CounterMap to a new file and improve encapsulation

be9f97f

lukesandberg force-pushed the 01-09-task-storage-macro-infrastructure branch from ab54eaf to be9f97f Compare January 15, 2026 02:37

lukesandberg commented Jan 15, 2026

View reviewed changes

suppress dead_code in the new module too, sigh

54be541

lukesandberg merged commit 02f8ba6 into canary Jan 15, 2026
166 checks passed

lukesandberg deleted the 01-09-task-storage-macro-infrastructure branch January 15, 2026 03:47

feat: add TaskStorage derive macro and schema infrastructure #88338

feat: add TaskStorage derive macro and schema infrastructure #88338

Conversation

lukesandberg commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Uh oh!

nextjs-bot commented Jan 9, 2026

Uh oh!

nextjs-bot commented Jan 9, 2026

Uh oh!

lukesandberg commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nextjs-bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests Passed

Uh oh!

nextjs-bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stats from current PR

✅ No significant changes detected

⚡ Dev Server

📦 Dev Server (Webpack)

⚡ Production Builds

📦 Production Builds (Webpack)

Bundle Sizes

⚡ Turbopack

📦 Webpack

🔄 Shared (bundler-independent)

Uh oh!

codspeed-hq bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging this PR will not alter performance

Summary

Footnotes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bgw Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

lukesandberg Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukesandberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lukesandberg commented Jan 15, 2026

Merge activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lukesandberg commented Jan 9, 2026 •

edited

Loading

lukesandberg commented Jan 9, 2026 •

edited

Loading

nextjs-bot commented Jan 9, 2026 •

edited

Loading

nextjs-bot commented Jan 9, 2026 •

edited

Loading

codspeed-hq bot commented Jan 9, 2026 •

edited

Loading