Skip to content

Use wit-deps to manage all wit dependencies#8928

Closed
iawia002 wants to merge 1 commit intobytecodealliance:mainfrom
iawia002:wit-deps
Closed

Use wit-deps to manage all wit dependencies#8928
iawia002 wants to merge 1 commit intobytecodealliance:mainfrom
iawia002:wit-deps

Conversation

@iawia002
Copy link
Copy Markdown
Contributor

This patch uses the wit-deps package to manage all wit dependencies (except wasi-nn for now):

# TODO: the in-tree `wasi-nn` implementation does not yet fully support the
# latest WIT specification on `main`. To create a baseline for moving forward,
# the in-tree WIT incorporates some but not all of the upstream changes. This
# TODO can be removed once the implementation catches up with the spec.

I believe this is a more understandable and automated approach compared to the bash script, as it will automatically synchronize these wit dependencies during the build process.

@iawia002 iawia002 requested review from a team as code owners July 10, 2024 04:33
@iawia002 iawia002 requested review from fitzgen and removed request for a team July 10, 2024 04:33
@iawia002 iawia002 force-pushed the wit-deps branch 2 times, most recently from 6ca9bc7 to 2f30e90 Compare July 10, 2024 04:51
@github-actions github-actions bot added the wasi Issues pertaining to WASI label Jul 10, 2024
@alexcrichton
Copy link
Copy Markdown
Member

Thanks for the PR! While I don't disagree that this is better than bash in terms of readability this has a downside of depending on wit-deps at build time which is a bit of a hefty dependency to add to the build. For now I think it would be best to keep the same style of build where the wit files are checked in CI to be correct but otherwise there's no extra build dependencies for downstream users.

@juntyr
Copy link
Copy Markdown
Contributor

juntyr commented Jul 10, 2024

Thanks for the PR! While I don't disagree that this is better than bash in terms of readability this has a downside of depending on wit-deps at build time which is a bit of a hefty dependency to add to the build. For now I think it would be best to keep the same style of build where the wit files are checked in CI to be correct but otherwise there's no extra build dependencies for downstream users.

What I’m doing in a project of mine at the moment is to use wit-deps in a test to check that everything is up to date (but to also keep the wit field checked in … well technically my solution uses git submodules to avoid checking the files in directly, but that doesn’t matter for the test) - this makes the dependency a dev-dependency only so that it should not burden any downstream users

@fitzgen
Copy link
Copy Markdown
Member

fitzgen commented Jul 10, 2024

Seconding what Alex said: we should run this in CI, not on every single build of the wasmtime crate.

@fitzgen fitzgen removed their request for review July 10, 2024 17:28
@iawia002
Copy link
Copy Markdown
Contributor Author

Thank all for your feedback. I understand your concerns and have made some improvements. I will no longer use build.rs to update dependencies. The process will remain as it is now, but developers will need to install the wit-deps CLI tool to update wit files locally.

Going back to the original reason why I wanted to make this change:

  1. I noticed that the current bash script keeps the dependencies of the wasi and wasi-http crates in sync, but in fact, the wasi crate does not need the http world:

    wasmtime::component::bindgen!({
    path: "wit",
    world: "wasi:cli/command",

    In the bash script, achieving this differentiation requires more code. If this patch is accepted, I will submit another PR to do some cleanup.

  2. When I tried to implement wasi-runtime-config, since it is still a draft version, its tag is different from cli and http etc. To manage its wit dependencies, I had to add some code to the bash script to download files individually. If there are more different versions of wit dependencies in the future, the bash script will become very complex.

PTAL if this change is reasonable.

@alexcrichton
Copy link
Copy Markdown
Member

Functionally this is much better, thanks! Personally though I'd at least personally prefer to stick to the bash script for now. I'm worried to pick up another tool as part of our build which much of us don't understand and it's not buying us much over a bash script. I don't disagree that the bash script is going to get a bit more complicated over time but I think the complexity is relatively manageable. For example it's not really an issue that HTTP WIT files are part of the wasmtime-wasi crate. Adding support for downloading another version of things to a different location shouldn't greatly increase the complexity of the script either.

Some other reasons I'd prefer to stick to a script for now:

  • Installation of wit-deps is yet-another-tool to manage in CI. For example cargo install here doesn't use --locked to prevent errors over time with mistakes in publishing to crates.io. It also doesn't specify a version to install so we'd be susceptible to breakage in terms of if wit-depschanges. This is all fixable but is adding more overhead relative to a script.
  • We can't exclusively use wit-deps because wasi-nn doesn't support it. That means we still need a script to do things. Additionally the script is required to matter what to pass the right arguments to wit-deps.

Overall I think it may just not be the right time to switch away from bash scripts. If the shell script could go away entirely I think that's the right time to migrate to a different tool, but I don't think that we're there yet.

@iawia002 iawia002 deleted the wit-deps branch July 18, 2024 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

wasi Issues pertaining to WASI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants