participating: @tschneidereit, @Mark-Simulacrum, @davidtwco, @weihanglo, @jlizen, @spastorino, @alexcrichton (async: @dtolnay)
High level approach
Say in my proc macro, this can be wasm compiled.
At the top level, when you build the project, opt in to "compile my macros to wasm".
Rustc actually needs to load the proc macro to do something, has some wasm runtime inside of rustc, calls into wasm, does the proc macro business, transfers tokens
More concretely (alex's opinion), wasm is a component, not a core module (wasi p2/p3), wasi aspect is more future compatible, but today you don't get fs, network, you can only print for now. Personally feels wasmtime is a good fit, licensing is fine.
Scope
The v0 goal is ONLY determinism related, we're not ignoring the build system security, those aspects exist, but the goal is around determinism/cachability, and only for the easy cases.
North star for initial scope is make it work for serde, but we suggest using wasi in part to make it work for eg sqlx with i/o or whatever else, reasons besides determinism, etc.
Alex glad to help with technical implementation planning, advise.
Maybe 2000-ish line rust file in rustc.
Wasmtime
Alex: can make sure wasmtime component is smaller than needed. Roughly 3-4 mb. (mark: not a problem)
Platform support: JI optimising compiler, cranelift for the big platform, for the long tail we have interpreter. Licensing: LLVM's license (apache w/ llvm)
Wasi is nice because you get more community libraries, will have more support from Alex.
Wasmtime does haver support now for self-hosted debugger that is compiled to a component, has access to a debugging api, etc, could be used to provide a good debugging experience for proc macros (and eventually build scripts) in theory - not initial scope.
Don't necessarily need to scope the goal to using wasmtime, could definitely in terms of requirements and see who can meet them, and wasmtime is certainly going to be able to but others could have the opportunity.
Wasmtime does have a non-trival tree, including tokio. No hyper, rustls, network dependencies. Cargo does already pull in futures (and potentially tokio in future). This might end up the limiting factor of platform support in short term (though existing systems will be kept in place for these targets). Till feels that there would be an appetite for investments in Akamai and closing gaps to broadening support.
Wasmtime supply chain security: large dependency tree, but using cargo-vet extensively and have made a lot of investments in that area. Mark: currently better postured than rustc
Wasm does release a new major version every month, unconditionally, every 12 months LTS with 2-year support (wasmtime 24, 36 are currently LTS). That gets you security backfixes (along with n-1, but that is only 2 months). Rustc could stick to LTS, mostly for security backfixes, or just use whatever is latest on shipping. There is an opportunity to tweak release cycles to align with rust, who knows.
Akamai would be interested in helping with upkeep, alex (personal capacity even if not work capacity) happy to review / own maintenance of the wasm integration in rustc.
Platform/target/stability
Perhaps don't even need to support all platforms.
For targets: visible to wasm via cfgs? Possible for wasm target, though want to be careful about forcing user agent probing
This is all unstable for now, get it in to get it running. If it becomes stable, you must pair wasm binary with the compiler. True today but would be a consideration if you ever can pre-compile (not in scope). All the apis are also virtualizable so we can always have wasm wrapper to adapt to other targets.
Cargo
Initial funding is from meta who uses bazel. So MVP scope is rustc-only. But, most participants in this discussion need it to work for Cargo as well.
For cargo, it does need to pass information to rustc saying that dependencies are wasm for proc macros, even if footprint is fairly limited. Need a conversation with team members than whlo@ to explore further to make sure we don't back ourselves into a future corner. Come stabilization time, might not need to actually share specific target triples with rustc, just say, "this is a proc macro that compiles to wasm, do what you want with it". Could take a similar approach to artifact dependencies.
Testing
Difficult to test in rustc because the build system would need to build wasm to validate. We do have some dependencies in rustc to already have some tests in validating wasm. Might be not so bad in CI but tough for contributors.
Performance
For performance to be fast, ultimately will want to cache the c-wasm artifact that wasmtime will generate from the wasm artifact. Start off by default and can enable if it becomes a concern. Reason to wait is mostly that is pre-compiled c wasms are very unstable, and we do want to leave door open for pre-compiled wasms, swappable runtimes, etc, anyway might be solvable in different ways.
Bigger performance gains in short term would be through improving cachability of artifacts in build systems more broadly. Though, it would make the benchmarks in rustc CI look potentially worse.
Mark: we would probably eventually do PGO + bolt for this (though maybe not bolt due to binary size)
Risks
A lot of wasm code out there assumes that it is wasm-32 and that there are web apis available, and wasi doesn't have it. People WILL want access to these APIs and that should be considered as future scope.
Future / open questions
Eventually build scripts as wasm as well. Entirely controlled by Cargo, but we are saying for initial scope rustc would embed the wasm runtime for proc macros, so what is the relationship?
In theory, you can invoke wasmtime directly but that infrastructure doesn't exist. Stabilizing some sort of cli interface in rustc might work as well. Though then we have rerun problems. Also want to be careful about re-inventing in wasmtime api. Could also ship a a build-script-runtime ie wastime binary directly, it is only 3mb.
participating: @tschneidereit, @Mark-Simulacrum, @davidtwco, @weihanglo, @jlizen, @spastorino, @alexcrichton (async: @dtolnay)
High level approach
Say in my proc macro, this can be wasm compiled.
At the top level, when you build the project, opt in to "compile my macros to wasm".
Rustc actually needs to load the proc macro to do something, has some wasm runtime inside of rustc, calls into wasm, does the proc macro business, transfers tokens
More concretely (alex's opinion), wasm is a component, not a core module (wasi p2/p3), wasi aspect is more future compatible, but today you don't get fs, network, you can only print for now. Personally feels wasmtime is a good fit, licensing is fine.
Scope
The v0 goal is ONLY determinism related, we're not ignoring the build system security, those aspects exist, but the goal is around determinism/cachability, and only for the easy cases.
North star for initial scope is make it work for serde, but we suggest using wasi in part to make it work for eg sqlx with i/o or whatever else, reasons besides determinism, etc.
Alex glad to help with technical implementation planning, advise.
Maybe 2000-ish line rust file in rustc.
Wasmtime
Alex: can make sure wasmtime component is smaller than needed. Roughly 3-4 mb. (mark: not a problem)
Platform support: JI optimising compiler, cranelift for the big platform, for the long tail we have interpreter. Licensing: LLVM's license (apache w/ llvm)
Wasi is nice because you get more community libraries, will have more support from Alex.
Wasmtime does haver support now for self-hosted debugger that is compiled to a component, has access to a debugging api, etc, could be used to provide a good debugging experience for proc macros (and eventually build scripts) in theory - not initial scope.
Don't necessarily need to scope the goal to using wasmtime, could definitely in terms of requirements and see who can meet them, and wasmtime is certainly going to be able to but others could have the opportunity.
Wasmtime does have a non-trival tree, including tokio. No hyper, rustls, network dependencies. Cargo does already pull in futures (and potentially tokio in future). This might end up the limiting factor of platform support in short term (though existing systems will be kept in place for these targets). Till feels that there would be an appetite for investments in Akamai and closing gaps to broadening support.
Wasmtime supply chain security: large dependency tree, but using cargo-vet extensively and have made a lot of investments in that area. Mark: currently better postured than rustc
Wasm does release a new major version every month, unconditionally, every 12 months LTS with 2-year support (wasmtime 24, 36 are currently LTS). That gets you security backfixes (along with n-1, but that is only 2 months). Rustc could stick to LTS, mostly for security backfixes, or just use whatever is latest on shipping. There is an opportunity to tweak release cycles to align with rust, who knows.
Akamai would be interested in helping with upkeep, alex (personal capacity even if not work capacity) happy to review / own maintenance of the wasm integration in rustc.
Platform/target/stability
Perhaps don't even need to support all platforms.
For targets: visible to wasm via cfgs? Possible for wasm target, though want to be careful about forcing user agent probing
This is all unstable for now, get it in to get it running. If it becomes stable, you must pair wasm binary with the compiler. True today but would be a consideration if you ever can pre-compile (not in scope). All the apis are also virtualizable so we can always have wasm wrapper to adapt to other targets.
Cargo
Initial funding is from meta who uses bazel. So MVP scope is rustc-only. But, most participants in this discussion need it to work for Cargo as well.
For cargo, it does need to pass information to rustc saying that dependencies are wasm for proc macros, even if footprint is fairly limited. Need a conversation with team members than whlo@ to explore further to make sure we don't back ourselves into a future corner. Come stabilization time, might not need to actually share specific target triples with rustc, just say, "this is a proc macro that compiles to wasm, do what you want with it". Could take a similar approach to artifact dependencies.
Testing
Difficult to test in rustc because the build system would need to build wasm to validate. We do have some dependencies in rustc to already have some tests in validating wasm. Might be not so bad in CI but tough for contributors.
Performance
For performance to be fast, ultimately will want to cache the c-wasm artifact that wasmtime will generate from the wasm artifact. Start off by default and can enable if it becomes a concern. Reason to wait is mostly that is pre-compiled c wasms are very unstable, and we do want to leave door open for pre-compiled wasms, swappable runtimes, etc, anyway might be solvable in different ways.
Bigger performance gains in short term would be through improving cachability of artifacts in build systems more broadly. Though, it would make the benchmarks in rustc CI look potentially worse.
Mark: we would probably eventually do PGO + bolt for this (though maybe not bolt due to binary size)
Risks
A lot of wasm code out there assumes that it is wasm-32 and that there are web apis available, and wasi doesn't have it. People WILL want access to these APIs and that should be considered as future scope.
Future / open questions
Eventually build scripts as wasm as well. Entirely controlled by Cargo, but we are saying for initial scope rustc would embed the wasm runtime for proc macros, so what is the relationship?
In theory, you can invoke wasmtime directly but that infrastructure doesn't exist. Stabilizing some sort of cli interface in rustc might work as well. Though then we have rerun problems. Also want to be careful about re-inventing in wasmtime api. Could also ship a a build-script-runtime ie wastime binary directly, it is only 3mb.