Preload python packages in workerd like in EW.#4190

Merged

dom96 merged 1 commit intomainfrom

dominik/preload-packages-workerd

Jun 19, 2025

Contributor

dom96 commented May 22, 2025 •

edited

Loading

This PR was implemented with the help of Windsurf.

This PR changes how workerd downloads Python packages. Instead of downloading them inside the worker using fetch, the packages are downloaded prior to isolate startup. This allows workerd to share the same startup code path as EW.

There are two left over tasks after this PR, but which are too big for this PR:

Tracing still requires IO, so we cannot init Python while tracing is on. It would be nice to avoid this, though I'm not sure how realistic that is.
The DiskCache class continues to exist, it is currently only used to save snapshot to disk for testing (and manual generation) purposes.

Test Plan

bazel run @workerd//src/workerd/server/tests/python:python-rpc_development@

dom96 force-pushed the dominik/preload-packages-workerd branch 7 times, most recently from 72b0646 to 4ea7502 Compare

May 23, 2025 14:53

jasnell reviewed

View reviewed changes

src/workerd/server/workerd-api.h Outdated Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/server/workerd-api.c++ Outdated Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/server/workerd-api.c++ Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/server/workerd-api.c++ Outdated Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/server/workerd-api.c++ Show resolved Hide resolved

dom96 force-pushed the dominik/preload-packages-workerd branch from 4ea7502 to 5e83346 Compare

May 23, 2025 15:10

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Outdated Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Outdated Show resolved Hide resolved

github-actions bot commented May 23, 2025 •

edited

Loading

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Show resolved Hide resolved

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Outdated Show resolved Hide resolved

dom96 force-pushed the dominik/preload-packages-workerd branch from 5e83346 to 6ce3338 Compare

May 23, 2025 15:20

jasnell reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Show resolved Hide resolved

dom96 force-pushed the dominik/preload-packages-workerd branch 3 times, most recently from 6bf70d6 to 83a00ab Compare

May 23, 2025 15:54

dom96 marked this pull request as ready for review

May 23, 2025 16:01

dom96 requested review from a team as code owners

May 23, 2025 16:01

dom96 requested a review from hoodmane

May 23, 2025 16:02

dom96 force-pushed the dominik/preload-packages-workerd branch 2 times, most recently from d8cad85 to 1e3c987 Compare

May 23, 2025 16:20

This comment was marked as duplicate.

Sign in to view

dom96 force-pushed the dominik/preload-packages-workerd branch from 11f7657 to ac4fdb0 Compare

June 12, 2025 15:56

This comment was marked as duplicate.

Sign in to view

dom96 force-pushed the dominik/preload-packages-workerd branch from ac4fdb0 to e2415ab Compare

June 12, 2025 17:55

This comment was marked as duplicate.

Sign in to view

dom96 force-pushed the dominik/preload-packages-workerd branch from e2415ab to f75e2ad Compare

June 13, 2025 10:45

graphite-app bot reviewed

View reviewed changes

src/workerd/api/pyodide/pyodide.c++ Outdated Show resolved Hide resolved

dom96 force-pushed the dominik/preload-packages-workerd branch 13 times, most recently from 4e4b0c6 to 6685441 Compare

June 16, 2025 18:21

hoodmane reviewed

View reviewed changes

src/workerd/server/tests/python/undefined-handler/server.mjs Outdated Show resolved Hide resolved

dom96 force-pushed the dominik/preload-packages-workerd branch 5 times, most recently from 1fba18a to bfd3c47 Compare

June 19, 2025 15:47


          Preload python packages in workerd like in EW.

c332b2f

dom96 force-pushed the dominik/preload-packages-workerd branch from bfd3c47 to c332b2f Compare

June 19, 2025 16:29

dom96 merged commit f63be49 into main

20 of 21 checks passed

dom96 deleted the dominik/preload-packages-workerd branch

June 19, 2025 17:13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

jasnell jasnell left review comments

hoodmane hoodmane approved these changes

danlapid danlapid left review comments

graphite-app[bot] graphite-app[bot] left review comments

+1 more reviewer

windsurf-bot[bot] windsurf-bot[bot] left review comments

Labels

None yet