We can also use this to insert a sleep into the Python main loop, so that the print statements from the Python loop and the Javascript loop are interleaved:

for(let i = 1; i < 100; i++){
  sleep(100*i).then(() => console.log("jsi", i));
}
f = py.runPython(`
def f():
  for i in range(100_000):
    if i % 100 == 0:
      print("pyi", i)
  import pytest # segfaults if pytest hasn't been imported before =(
f
`);
await f.callSyncifying({});

So the stack switching is extremely promising. It also seems likely that we will be able to make a version of Pyodide that uses stack switching if it is available but gracefully falls back if it is not available.

Thanks for the investigation @hoodmane! Indeed it looks quite promising.

with node v18.x

Nice. Since node v18 is now in an LTS state, I think it is okay to use new features of v18.

Apparently they completely rearranged this proposal though. It seems like it now requires wasm reference types to use, and they may have removed the WebAssembly.Suspender API. Wasm reference types have been available in Firefox since v79 July 2020, in Safari since v15.0 released in September 2021, and in Chrome since v96 released November 2021. So it would be nice not to use them. But also, there is currently no way to make C/C++ code that emits wasm reference types, the only documented way to use them is via assembly code.
emscripten-core/emscripten#15878

Both the backwards compatibility problem and the how-do-we-even-emit-code problem could be solved by hand coding the wrapper like what I did for createDyncallWrapper but this will be a pain:
https://github.com/emscripten-core/emscripten/pull/17328/files#diff-8bb085f397f4174fe93a8cf0cda27a12a2647a4afe8ce7116ed740935f5d1188

I will have to try rewriting this to use the newer approach and see if it fixes the segfaults...

See:
https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md

Okay, so this works great in node v19 with flags --experimental-wasm-type-reflection and --experimental-wasm-stack-switching or in chrome version 109.0.5384.0 (Official Build) dev with the Experimental WebAssembly Stack Switching flag set to enabled. I have two tests: one shows that we can allow background tasks to occur intermingled with synchronous Python. Another uses an import hook to download and install pytest when an import is attempted.

This feature magically transcends our most annoying limitations.

I think the trickiest design question is how to deal with mixing syncifying calls with non syncifying calls. Syncifying calls always return a promise. Non syncifying calls trap if a promise is syncified. It should be possible to set a flag on entry to the vm indicating whether it was a syncifying entry or a nonsyncifying one. A more sophisticated approach to reentrancy might be needed as well.

We should probably look into using the wasm binary toolkit to convert wat text source files to wasm. Then we can convert the wasm to base64, create javascript files which define the module binaries, and inject these javascript files with --pre-js.

@rth @ryanking13 this is ready to review. By splitting off a bunch of stuff, I've gotten it down to 408 lines added for features and 326 lines for tests which is more than a 2x decrease in size from before. The current state of this PR leaves out three things:

• ctypes integration
• stack state management
• Python state management

I have dealt with all three of these issues on local branches and will make more followup PRs after this is merged to fix them.

Thanks for doing this work! I'll review it tomorrow.

@rth okay to merge this?

Sorry for the delay @hoodmane . Yes, no objection on my part. Thanks for doing all this work!