Update Wasmtime for upcoming WIT changes #6390

alexcrichton · 2023-05-16T21:23:05Z

This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The main changes here are:

WIT syntax is updated with Proposal: Update WIT syntax/semantics for packages/dependencies/use and alter binary format for externnames WebAssembly/component-model#193
Generated bindings in the bindgen! macro have been updated to reflect the new structure of WIT.
The accepted component model binary format has been updated to account for changes.

This PR disables wasi-http tests and the on-by-default feature because the WIT syntax has been updated but the submodule containing the WITs has not been updated yet so there's no way to get that building temporarily. Once that's updated then this can be reenabled.

alexcrichton · 2023-05-16T21:23:41Z

I'll note that I'm opening this as a draft as I'm working my way up the stack. My goal is to prove out bytecodealliance/wasm-tools#1027 and once everything is ready have a day I spend landing the various changes.

This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The main changes here are: * WIT syntax is updated with WebAssembly/component-model#193 * Generated bindings in the `bindgen!` macro have been updated to reflect the new structure of WIT. * The accepted component model binary format has been updated to account for changes. This PR disables wasi-http tests and the on-by-default feature because the WIT syntax has been updated but the submodule containing the WITs has not been updated yet so there's no way to get that building temporarily. Once that's updated then this can be reenabled.

Don't store duplicates with the rest of the WASI WIT files we have.

alexcrichton · 2023-05-25T21:10:32Z

Ok I've now updated this for the final set of merged changes to wasm-tools, which aren't yet published. The major changes in this PR are now:

All wasm-tools deps are updated (currently sitting at git deps to my "bump" branch)
WIT syntax is changing, as well as the binary format of components
All WIT files vendored in this repository are updated to the new syntax. Notably the wasi-http WITs are not updated because they're a submodule. This means that all wasi-http-related tests and support are disabled.
There is now only one copy of WIT for WASI, and all other duplicate copies are deleted.
All of the preview2 implementation has been updated for the new hierarchy of modules created by wasmtime::component::bindgen!

One interesting aspect of this is that there's a temporary git dependency on wit-bindgen. I'm not actually sure how to get rid of that because the adapter requires wit-bindgen and it's not easy to remove that. This is somewhat ok though since we don't publish anything with wit-bindgen so I'm hoping it's ok to have a temporary state of affairs with a git dep here before I can go use this PR to update wit-bindgen to make a release there.

Make it a more "official" location which is also less likely to be accidentally deleted in the future.

rvolosatovs · 2023-05-31T14:49:56Z

Seems like a component using latest WIT interfaces generated with wit-component 0.11.0 fails to compile with:

    0: failed to parse WebAssembly module
    1: unsupported component version: 0xc (at offset 0x0)

Originating somewhere in

wasmtime/crates/environ/src/component/translate.rs

Lines 328 to 377 in c044790

    
               pub fn translate( 
        
                   mut self, 
        
                   component: &'data [u8], 
        
               ) -> Result<( 
        
                   Component, 
        
                   PrimaryMap<StaticModuleIndex, ModuleTranslation<'data>>, 
        
               )> { 
        
                   // First up wasmparser is used to actually perform the translation and 
        
                   // validation of this component. This will produce a list of core wasm 
        
                   // modules in addition to components which are found during the 
        
                   // translation process. When doing this only a `Translation` is created 
        
                   // which is a simple representation of a component. 
        
                   let mut remaining = component; 
        
                   loop { 
        
                       let payload = match self.parser.parse(remaining, true)? { 
        
                           Chunk::Parsed { payload, consumed } => { 
        
                               remaining = &remaining[consumed..]; 
        
                               payload 
        
                           } 
        
                           Chunk::NeedMoreData(_) => unreachable!(), 
        
                       }; 
        
                       match self.translate_payload(payload, component)? { 
        
                           Action::KeepGoing => {} 
        
                           Action::Skip(n) => remaining = &remaining[n..], 
        
                           Action::Done => break, 
        
                       } 
        
                   } 
        
                   assert!(remaining.is_empty()); 
        
                   assert!(self.lexical_scopes.is_empty()); 
        
                   // ... after translation initially finishes the next pass is performed 
        
                   // which we're calling "inlining". This will "instantiate" the root 
        
                   // component, following nested component instantiations, creating a 
        
                   // global list of initializers along the way. This phase uses the simple 
        
                   // initializers in each component to track dataflow of host imports and 
        
                   // internal references to items throughout a component at compile-time. 
        
                   // The produce initializers in the final `Component` are intended to be 
        
                   // much simpler than the original component and more efficient for 
        
                   // Wasmtime to process at runtime as well (e.g. no string lookups as 
        
                   // most everything is done through indices instead). 
        
                   let mut component = inline::run( 
        
                       &self.types, 
        
                       &self.result, 
        
                       &self.static_modules, 
        
                       &self.static_components, 
        
                   )?; 
        
                   self.partition_adapter_modules(&mut component); 
        
                   Ok((component.finish(), self.static_modules)) 
        
               }

Refs wasmCloud/wasmCloud#347

alexcrichton · 2023-05-31T14:51:47Z

Can you confirm that wit-component 0.11.0 was used? The version 0xc is the "old" version of components and the upgraded versions should be 0xd, so it may be the case that wit-component 0.11.0 wasn't used by accident?

rvolosatovs · 2023-05-31T14:58:27Z

Can you confirm that wit-component 0.11.0 was used? The version 0xc is the "old" version of components and the upgraded versions should be 0xd, so it may be the case that wit-component 0.11.0 wasn't used by accident?

I'll try to minimize this later today

rvolosatovs · 2023-05-31T17:52:01Z

Can you confirm that wit-component 0.11.0 was used? The version 0xc is the "old" version of components and the upgraded versions should be 0xd, so it may be the case that wit-component 0.11.0 wasn't used by accident?

false alarm, it was an outdated transitive dependency. Can confirm that everything works as expected. Thanks for prompt response!

* Update Wasmtime for upcoming WIT changes This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The main changes here are: * WIT syntax is updated with WebAssembly/component-model#193 * Generated bindings in the `bindgen!` macro have been updated to reflect the new structure of WIT. * The accepted component model binary format has been updated to account for changes. This PR disables wasi-http tests and the on-by-default feature because the WIT syntax has been updated but the submodule containing the WITs has not been updated yet so there's no way to get that building temporarily. Once that's updated then this can be reenabled. * Update wasmtime-wasi crate with new WIT * Add wit-bindgen override for the updated version * Officially disable wasi-http tests/building * Move test-reactor WIT into the main WIT files Don't store duplicates with the rest of the WASI WIT files we have. * Remove adapter's copy of WIT files * Disable default features for wit-bindgen * Plumb disabling wasi-http tests a bit more * Fix reactor tests and adapter build * Remove no-longer-needed feature * Update adapter verification script * Back out some wasi-http hacks * Update vet and some dependency sources * Move where wit-bindgen comes from Make it a more "official" location which is also less likely to be accidentally deleted in the future.

These files aren't actually read by anything currently. They were added historically and a previous refactoring in bytecodealliance#6390 forgot to remove them. No tests or build process reads them so this deletes them to get them out of the way.

#6777) * Remove unused WIT files from Wasmtime These files aren't actually read by anything currently. They were added historically and a previous refactoring in #6390 forgot to remove them. No tests or build process reads them so this deletes them to get them out of the way. * Update dependencies on wasm-tools crates. This commit updates the deps on the wasm-tools family of crates to bring in a few fixes for WIT/component-related things. Primarily though this brings in an update to the component model where empty types are now disallowed. Some tests using empty types were adjusted to use non-empty types, but many tests were also simply removed as they existed to test what would happen with empty types which now no longer needs to be tested. * Update `stream-error` in preview2 Add a `dummy` field to make it a non-empty structure. It's expected that this will change to something else more "official" in the future, but for now this is here to keep everything compiling. * Update component fuzzing to avoid empty types Empty types are no longer valid * Update crates/wasi/wit/deps/io/streams.wit Co-authored-by: Peter Huene <peter@huene.dev> --------- Co-authored-by: Peter Huene <peter@huene.dev>

bytecodealliance#6777) * Remove unused WIT files from Wasmtime These files aren't actually read by anything currently. They were added historically and a previous refactoring in bytecodealliance#6390 forgot to remove them. No tests or build process reads them so this deletes them to get them out of the way. * Update dependencies on wasm-tools crates. This commit updates the deps on the wasm-tools family of crates to bring in a few fixes for WIT/component-related things. Primarily though this brings in an update to the component model where empty types are now disallowed. Some tests using empty types were adjusted to use non-empty types, but many tests were also simply removed as they existed to test what would happen with empty types which now no longer needs to be tested. * Update `stream-error` in preview2 Add a `dummy` field to make it a non-empty structure. It's expected that this will change to something else more "official" in the future, but for now this is here to keep everything compiling. * Update component fuzzing to avoid empty types Empty types are no longer valid * Update crates/wasi/wit/deps/io/streams.wit Co-authored-by: Peter Huene <peter@huene.dev> --------- Co-authored-by: Peter Huene <peter@huene.dev>

) * Update Wasmtime for upcoming WIT changes This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The main changes here are: * WIT syntax is updated with WebAssembly/component-model#193 * Generated bindings in the `bindgen!` macro have been updated to reflect the new structure of WIT. * The accepted component model binary format has been updated to account for changes. This PR disables wasi-http tests and the on-by-default feature because the WIT syntax has been updated but the submodule containing the WITs has not been updated yet so there's no way to get that building temporarily. Once that's updated then this can be reenabled. * Update wasmtime-wasi crate with new WIT * Add wit-bindgen override for the updated version * Officially disable wasi-http tests/building * Move test-reactor WIT into the main WIT files Don't store duplicates with the rest of the WASI WIT files we have. * Remove adapter's copy of WIT files * Disable default features for wit-bindgen * Plumb disabling wasi-http tests a bit more * Fix reactor tests and adapter build * Remove no-longer-needed feature * Update adapter verification script * Back out some wasi-http hacks * Update vet and some dependency sources * Move where wit-bindgen comes from Make it a more "official" location which is also less likely to be accidentally deleted in the future. * Don't document wasi-http-tests

github-actions bot added cranelift Issues related to the Cranelift code generator cranelift:wasm labels May 16, 2023

alexcrichton force-pushed the wit-changes branch 2 times, most recently from cf306c4 to ea497d8 Compare May 17, 2023 16:19

alexcrichton mentioned this pull request May 17, 2023

Update WIT and binary import/export syntax WebAssembly/component-model#198

Merged

alexcrichton force-pushed the wit-changes branch from 46031a2 to 1f8cf72 Compare May 17, 2023 19:12

alexcrichton mentioned this pull request May 18, 2023

Update wit-bindgen for upcoming WIT changes bytecodealliance/wit-bindgen#580

Merged

alexcrichton added 10 commits May 25, 2023 12:08

Update wasmtime-wasi crate with new WIT

507a23a

Add wit-bindgen override for the updated version

362ade0

Officially disable wasi-http tests/building

0716278

Move test-reactor WIT into the main WIT files

cb9c266

Don't store duplicates with the rest of the WASI WIT files we have.

Remove adapter's copy of WIT files

75e1529

Disable default features for wit-bindgen

732feff

Plumb disabling wasi-http tests a bit more

a8dea2c

Fix reactor tests and adapter build

067f535

Merge remote-tracking branch 'origin/main' into wit-changes

6765338

alexcrichton force-pushed the wit-changes branch from 6054730 to 6765338 Compare May 25, 2023 21:05

alexcrichton marked this pull request as ready for review May 25, 2023 21:07

alexcrichton requested review from a team as code owners May 25, 2023 21:07

alexcrichton requested review from elliottt and jameysharp and removed request for a team May 25, 2023 21:07

alexcrichton removed request for a team and elliottt May 25, 2023 21:10

Mossaka mentioned this pull request May 26, 2023

Update to the upcoming WITs spinframework/spin#1544

Closed

alexcrichton added 3 commits May 26, 2023 07:04

Merge remote-tracking branch 'origin/main' into wit-changes

ae625d1

Update vet and some dependency sources

7ff4d0a

Move where wit-bindgen comes from

d1c3782

Make it a more "official" location which is also less likely to be accidentally deleted in the future.

alexcrichton enabled auto-merge May 26, 2023 14:21

alexcrichton added this pull request to the merge queue May 26, 2023

Don't document wasi-http-tests

02c72af

alexcrichton removed this pull request from the merge queue due to a manual request May 26, 2023

alexcrichton enabled auto-merge May 26, 2023 15:01

alexcrichton added this pull request to the merge queue May 26, 2023

github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch May 26, 2023

Merge remote-tracking branch 'origin/main' into wit-changes

68d7f30

alexcrichton enabled auto-merge May 26, 2023 16:07

Merge remote-tracking branch 'origin/main' into wit-changes

9df449e

alexcrichton added this pull request to the merge queue May 26, 2023

Merged via the queue into bytecodealliance:main with commit 41417d9 May 26, 2023

alexcrichton deleted the wit-changes branch May 26, 2023 17:13

rvolosatovs mentioned this pull request May 31, 2023

build: update WIT dependencies wasmCloud/wasmCloud#347

Merged

10 tasks

rvolosatovs mentioned this pull request May 31, 2023

fix downstream cargo vendor usage for component adapter crate #6486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update Wasmtime for upcoming WIT changes #6390

Update Wasmtime for upcoming WIT changes #6390

Uh oh!

alexcrichton commented May 16, 2023

Uh oh!

alexcrichton commented May 16, 2023

Uh oh!

alexcrichton commented May 25, 2023

Uh oh!

Uh oh!

rvolosatovs commented May 31, 2023 •

edited

Loading

Uh oh!

alexcrichton commented May 31, 2023

Uh oh!

rvolosatovs commented May 31, 2023

Uh oh!

rvolosatovs commented May 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Update Wasmtime for upcoming WIT changes #6390

Update Wasmtime for upcoming WIT changes #6390

Uh oh!

Conversation

alexcrichton commented May 16, 2023

Uh oh!

alexcrichton commented May 16, 2023

Uh oh!

alexcrichton commented May 25, 2023

Uh oh!

Uh oh!

rvolosatovs commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexcrichton commented May 31, 2023

Uh oh!

rvolosatovs commented May 31, 2023

Uh oh!

rvolosatovs commented May 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rvolosatovs commented May 31, 2023 •

edited

Loading