Skip to content

Increase external memory adjustment for fetch() and other subrequests#6113

Merged
jasnell merged 2 commits intomainfrom
harris/EW-10528-increase-external-memory-adjustment-for-fetch
Feb 24, 2026
Merged

Increase external memory adjustment for fetch() and other subrequests#6113
jasnell merged 2 commits intomainfrom
harris/EW-10528-increase-external-memory-adjustment-for-fetch

Conversation

@harrishancock
Copy link
Copy Markdown
Collaborator

This PR:

  • Adds a new autogate, INCREASE_EXTERNAL_MEMORY_ADJUSTMENT_FOR_FETCH.
  • Removes an old autogate, FETCH_REQUEST_MEMORY_ADJUSTMENT.
  • Increases the external memory adjustment for fetch()es from 3 KiB to 8 KiB when the new autogate is on.
  • Moves the external memory adjustment to IoContext::getSubrequestNoChecks() so that it covers all subrequests, not just fetch() subrequests.
  • Adds the same external memory adjustment to Cache API subrequests.

In testing, I have discovered that the real C++ heap overhead of fetch() is 8 KiB if it ends up being queued in our concurrent connection limiter, and 32 KiB if it manages to establish a connection. So, this change brings the external memory adjustment closer to the actual unaccounted-for memory, regardless of whether or not the fetch() is queued. Also note that other factors, such as header size, could increase the overhead significantly higher, but probably don't much influence the mean or median overhead.

Moving the external memory adjustment from just fetch() to all subrequests is probably the most consequential part of this change, and worth discussing.

The dilemma is this: We'd like to relax the concurrent connection limiting behavior and rely solely on heap pressure to force the Worker to keep its concurrent subrequest connection count under control. So, it makes sense to apply this external memory adjustment to all subrequest types which end up establishing a network connection, not just fetch(). But, we can't know if a subrequest will actually need to establish a network connection until much deeper in our internal integration, where it would be an abstraction leak to add an external memory adjustment. This is already even true for fetch() itself -- it may invoke a Service Binding and not go over the network, after all.

I think expanding the external memory adjustment to cover all subrequest types is the right thing to do largely because all of the alternatives seem worse: We don't want to leave any potential network subrequests unaccounted for; we don't want maintain a a brittle, whackamole solution early in the call graph (close to fetch(), etc.); and we don't want to implement an abstraction leak later in the call graph (close to the actual network connection).

In a more positive light, I would also argue that applying the memory adjustment to subrequests unconditionally has the nice property that it applies pressure on the root originator of pressure in the system: if a Worker issues 100 fetch()es to 100 different service bindings, each of which then establishes one network connection, the root Worker receives the majority of the memory pressure to constrain its concurrency.

@harrishancock harrishancock requested review from a team as code owners February 19, 2026 14:37
@harrishancock harrishancock force-pushed the harris/EW-10528-increase-external-memory-adjustment-for-fetch branch from 9b050f9 to b75c6d2 Compare February 19, 2026 14:43
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Feb 19, 2026

Codecov Report

❌ Patch coverage is 88.23529% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.58%. Comparing base (0a7b469) to head (ffed0c5).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
src/workerd/io/io-context.c++ 85.71% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #6113   +/-   ##
=======================================
  Coverage   70.58%   70.58%           
=======================================
  Files         409      409           
  Lines      109672   109685   +13     
  Branches    18070    18074    +4     
=======================================
+ Hits        77408    77425   +17     
+ Misses      21445    21443    -2     
+ Partials    10819    10817    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@harrishancock harrishancock force-pushed the harris/EW-10528-increase-external-memory-adjustment-for-fetch branch 2 times, most recently from 22119db to 0974e57 Compare February 23, 2026 14:17
@codspeed-hq

This comment was marked as outdated.

harrishancock and others added 2 commits February 24, 2026 11:04
Add INCREASE_EXTERNAL_MEMORY_ADJUSTMENT_FOR_FETCH autogate that increases
the external memory adjustment for fetch() from 3 KiB to 8 KiB. This
brings it closer to the actual C++ memory overhead.

Also removes the fully-rolled-out FETCH_REQUEST_MEMORY_ADJUSTMENT autogate,
making the 3 KiB adjustment unconditional by default.

Release note: None

Co-Authored-By: Claude <noreply@anthropic.com>
When the INCREASE_EXTERNAL_MEMORY_ADJUSTMENT_FOR_FETCH autogate is
enabled, apply an 8 KiB external memory adjustment to all
concurrency-limited subrequests (fetch, KV, R2, Cache, Queues,
Hyperdrive, TCP, WebSocket), not just fetch(). This is done centrally
in IoContext::getSubrequestNoChecks() and IoContext::getCacheClient()
using IoContext::currentLock to obtain the jsg::Lock without requiring
any signature changes.

When the autogate is off, the existing 3 KiB adjustment remains in
fetchImplNoOutputLock() for fetch() only, preserving the old behavior.

Co-Authored-By: Claude <noreply@anthropic.com>
@harrishancock harrishancock force-pushed the harris/EW-10528-increase-external-memory-adjustment-for-fetch branch from 0974e57 to ffed0c5 Compare February 24, 2026 11:04
@jasnell jasnell merged commit 1a5413f into main Feb 24, 2026
22 of 23 checks passed
@jasnell jasnell deleted the harris/EW-10528-increase-external-memory-adjustment-for-fetch branch February 24, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants