Yes, that makes sense. However, to support this model, we need the test-programs crate to have WIT definition files of all the WASI APIs. For example, wasi-runtime-config is a very simple and standalone API, it doesn't depend on any other WASI APIs, nor is it depended upon by others. If we want to include both wasi-cli and wasi-runtime-config in a single generate!, their WIT files need to be in the same directory. I'm wondering if we could place the WIT files for all APIs in a separate directory, so that wasi, wasi-http, and test-programs can all directly use the WIT files from this directory. This would also simplify the vendor-wit.sh script.

Hope I didn't misunderstand your point.

The solution we've had for that is to have copies of WIT files in multiple locations. It's not a great solution but it's why wasmtime-wasi has all the WIT files for wasmtime-wasi-http despite not using them. Unfortunately we can't move everything into a single directory easily because when we publish these crates to crates.io that directory needs to be included so the crates can be built when downloaded from crates.io

because when we publish these crates to crates.io that directory needs to be included so the crates can be built when downloaded from crates.io

Oh right, since test programs don't need to be published to crates.io, I forgot to consider that. I'll make an update soon.

Could the WIT folder not be duplicated here? Since test-programs isn't published to crates.io it should be able to refer to the folder defined in the wasmtime-wasi or wasmtime-wasi-http crate

Nope, it has to, as I mentioned before, this change is mainly for the upcoming addition of APIs like wasi-runtime-config. These APIs are not dependent on other APIs and are not depended on by other APIs, so their WIT files will not be included in the wasi or wasi-http crates. We will need the test-programs to include the WIT files of all APIs.

And I plan to make some improvements to the vendor-wit script (pure bash script, no additional dependencies). Ultimately, each wasi API crate will only include the necessary WIT files. For example, wasi-runtime-config will only include one WIT dependency, and the implementation of wasi-cli will no longer include the WIT files for wasi-http.

It's a bit difficult to review this as it's setting up for future changes I at least personally have not yet seen or reviewed. In that sense it's difficult to review changes when only half the motivation is apparent, so I'd ask you bear with me in my questions and review feedback here.

Ideally we only have one copy of all WIT files in-tree. Even today it's a bummer we have one copy in wasmtime-wasi and one copy in wasmtime-wasi-http. How we manage WIT has changed over time but we've never settled on something everyone's happy with. The current status quo is the sort of "local maximum" where it's low-overhead to maintain and the management in CI is relatively simple.

In-tree there's also two main "styles" of sorts (not many data points to draw from) of managing WIT. One is wasmtime-wasi/wasmtime-wasi-http which are somewhat "tied at the hip". They're managed similarly and have the same WIT files. Another is wasi-nn which, like wasi-runtime-config, has no other WIT dependencies and is managed differently. Everything has a wit_bindgen::generate! call in the test-programs crate.

With the eventual goal of adding wasi-runtime-config, I would recommend one of two routes:

1. Follow the example of wasi-nn. This means that you'd add a few lines to the bash script to vendor wasi-runtime-config WIT files in a new directory. There'd be a new wit_bindgen::generate! call in test-programs as well. I don't think that it would require reorganizing things too much.
2. Refactor things before landing wasi-runtime-config to have a single generate! call in test-programs. That's sort of what this PR is doing but it's compromising on some of the initial design goals of our WIT management. I'll reiterate our WIT management is not great but we're also hoping to not make it worse (e.g. another copy of WIT in-tree).

With the (1) route it should be possible to add everything today without changes. With (2) I would recommend first refactoring the wit-bindgen crate itself, for example it would probably be best to enable something like this:

wit_bindgen::generate!({
    path: ["../wasmtime-wasi/wit", "../wasmtime-wasi-http/wit", "../wasi-nn/wit"],
    inline: "
        include wasi:cli/imports;
        include wasi:http/imports;
        include wasi:nn/imports;
    ",
});

I don't think that's fully supported in wit-bindgen today but it shouldn't be too too hard to add. That would enable not needing to copy around WIT files while still enabling a single generate! call. There's need to be a separate generate! call for wasi:http/proxy and other worlds used by the test suite as well.

Thanks for getting back to me, let's hold this for now.

My original intention for making these changes before implementing the wasi-runtime-config was to simplify the future PR and focus more on implementing the wasi-runtime-config API itself.

I was too fixated on getting these future tasks done beforehand. It seems I got the sequence of everything wrong. Let's go back to the beginning, after I have implemented the wasi-runtime-config, we can go back to those subsequent works.