✅ Deploy Preview for rspack canceled.

Built without sensitive environment variables

Would you please elaborate on "I find the main thread in nodejs doesn't exit if nodejs worker get stuck."? Is this problem exist when nodejs workers encounter some error or exist on every executions?

CodSpeed Performance Report

Merging #9895 will not alter performance

_{Comparing CPunisher:04-03-fix/wasi-execution-shutdown (8fc3b7d) with main (dfc27d8)}

Summary

✅ 11 untouched benchmarks

Would you please elaborate on "I find the main thread in nodejs doesn't exit if nodejs worker get stuck."? Is this problem exist when nodejs workers encounter some error or exist on every executions?

Here is an example. When you run node main.js, you will get stuck.

// main.js
const { Worker } = require('worker_threads');
const worker = new Worker('./worker.js')
console.log("main end")

// worker.js
console.log("worker start")
while (true) {}
console.log("worker end")

Now let me clarify the problems with rayon:

1. Rspack wasi (powered by emnapi) create threads by creating node workers
2. Although rayon falls back to sequential iteration in wasm, I find it still create its worker threads which lives in the whole lifetime of program. In normal native program, it's ok because the main thread doesn't wait for child threads without join.
3. Rolldown avoids using rayon by wasm shims: https://github.com/rolldown/rolldown/blob/main/crates/rolldown_utils/src/rayon.rs. It's very hard to apply it in rspack because some rayon iter api is not consistent with std iter api such as fold. We may need to do more to improve the wasm shims or write lots of #[cfg].

Although rayon falls back to sequential iteration in wasm, I find it still create its worker threads which lives in the whole lifetime of program.

That's weird. I haven't really dug into the implementation in WASI node. But I thought rayon tasks would be sent to worker_threads for execution instead of being executed in sequence.

In normal native program, it's ok because the main thread doesn't wait for child threads without join.

Is it possible to unref() all Workers? This should make the worker threads exit if main thread finishes its execution?

That's weird. I haven't really dug into the implementation in WASI node. But I thought rayon tasks would be sent to worker_threads for execution instead of being executed in sequence.

You could be right! Actually I know this from https://github.com/rayon-rs/rayon?tab=readme-ov-file#usage-with-webassembly, but I think it's very valuable to optimize in the future.

Is it possible to unref() all Workers? This should make the worker threads exit if main thread finishes its execution?

Thanks! I didn't know this api before. I quickly tried but didn't solve the problem. I will do more research about it.

It seems that worker.on('message', ...) will call worker.ref() again, see: nodejs/node#53036. A hacky way is adding worker.ref = () => {} to prevent workers being refed.