Add enough Windows functionality to run WASI tutorial#22
Add enough Windows functionality to run WASI tutorial#22sunfishcode merged 23 commits intoCraneStation:masterfrom kubkon:win-wasi-tutorial
Conversation
sunfishcode
left a comment
There was a problem hiding this comment.
Looks good so far! Here are a few drive-by comments -- I realize this code is work-in-progress, so if these aren't helpful right now, feel free to disregard :-).
src/sys/windows/host_impl.rs
Outdated
| len, | ||
| &mut host_nwritten, | ||
| std::ptr::null_mut(), | ||
| ) |
There was a problem hiding this comment.
Yep, that definitely needs to be handled. In fact, I've just now done some work in that direction. I've decided to move some of the unsafe logic into a separate crate, winx. My plan with this crate is to essentially imitate nix crate (to a degree of course).
Anyhow, now the calls to close a raw Win handle check for errors, and so does duplicate and writing the IoVec to a handle. Anyhow, for the latter, have a look here.
src/sys/windows/host_impl.rs
Outdated
| let buf = iovecs | ||
| .iter() | ||
| .find(|b| !b.as_slice().is_empty()) | ||
| .map_or(&[][..], |b| b.as_slice()); |
There was a problem hiding this comment.
Do you know if WSASend works on files? If not, should we iterate over the iovecs and call WriteFile on each individually?
There was a problem hiding this comment.
That's an excellent question. I've been wondering about that myself as well. The short answer is, I'm not sure. However, the current implementation you've highlighted here is taken verbatim from Rust nightly's implementation. In Rust, a call to IoSlice write_vectored eventually resolves into default_write_vectored.
In Rust, IoSlice is also a wrapper around WSABUF, however, it seems it's reserved only for file operations which I'm guessing would exclude sockets? Anyhow, I think it might be prudent to have a closer look at WSASend as well as the general implementation advocated by Rust and see what's what, and what to use. For the moment, I'll leave it as is, however, I'll come back to it when I get the WASI tutorial working. Having said that though, if you come up with something, then please fire away!
src/sys/windows/hostcalls/fs.rs
Outdated
| | host::__WASI_RIGHT_FD_WRITE | ||
| | host::__WASI_RIGHT_FD_ALLOCATE | ||
| | host::__WASI_RIGHT_FD_FILESTAT_SET_SIZE) | ||
| != 0; |
There was a problem hiding this comment.
Can the logic for determining if we need read and/or write here be factored into a utility function that can be shared with the unix impl?
There was a problem hiding this comment.
Ah, a great question! With the winx crate, I think the hostcalls will start to share more and more code between *nix and Windows, so we'll definitely have some room for refactoring here. Perhaps we could twist it slightly and have the hostcalls (as the public API entry) outside of host specific modules, and have them call host specific bits only? I'm wondering how messy would that be? Anyhow, definitely something on my radar! :-)
There was a problem hiding this comment.
Sharing code to this degree would be great, though not if it makes things more complicated, so we'll see where it goes :-). Factoring out the code for inferring read/write here would be especially nice because deciding exactly which rights determine this is tricky, and fairly likely to change as WASI evolves.
There was a problem hiding this comment.
I've factored out common functionality in #26. Have a look!
Oh no, please keep them coming! The more discussion we do early on, the better the output will be -- I'm a firm believer in that :-) |
Clean up closing and duplicating RawHandle
alexcrichton
left a comment
There was a problem hiding this comment.
@tschneidereit pointed me at this and I'm personally quite intrigued by all this so tried to leave some comments here and there!
One thing I was thinking is that it might be a good idea to reuse the standard library as much as possible. Within libstd (and throughout the Rust ecosystem) there's lots of trickiness in getting syscalls and such right, so it'd be great to be able to reuse all the the existing knowledge! Now this is more of a long-term thing of course in the sense that there's no need to get this perfect on the first try, but just something I was thinking!
src/sys/windows/fdentry.rs
Outdated
| // disk file: file, dir or disk device | ||
| let file = File::from_raw_handle(raw_handle); | ||
| let meta = file.metadata(); | ||
| std::mem::forget(file); |
There was a problem hiding this comment.
FWIW a helpful pattern I've found in the past is that instead of from_raw_* plus forget you can do:
let file = ManuallyDrop::new(File::from_raw_handle(raw_handle));
let meta = file.metadata();
// ...and the usage of ManuallyDrop means you don't have to call forget and can continue to use the File if necessary
There was a problem hiding this comment.
Nice one, thanks! That's a fantastic suggestion :-)
There was a problem hiding this comment.
Done! For the moment, I've decided to only include it in the bits pertaining this very PR (so Win only), whereas I'll re-apply the suggestion to *nix bits in a separate PR.
| } | ||
| } | ||
| winx::handle::close(self.raw_handle) | ||
| .unwrap_or_else(|e| eprintln!("FdObject::drop(): {}", e)) |
There was a problem hiding this comment.
FWIW we explicitly ignore errors in the Rust standard library for errors on close. We largely do that to mirror weirdness on Unix, but perhaps this should mirror that?
(or would it be possible to use a File from the standard library maybe?)
There was a problem hiding this comment.
Thanks for the pointer! So the reason we've got error handling here is essentially because I mirrored the implementation here from the corresponding *nix implementation in wasi-common. Having said that, it might indeed be obsolete :-)
(Hmm, perhaps we could use File instead. If we could, then, again, we could consolidate a fair amount of *nix and Win code, so it'd be amazing!)
There was a problem hiding this comment.
Ah yeah that makes sense (mirroring) and I don't want to try to pile on too much into one PR which is just trying to get things done!
| use std::os::windows::prelude::RawHandle; | ||
| use winapi::shared::minwindef::FALSE; | ||
|
|
||
| pub fn dup(old_handle: RawHandle) -> Result<RawHandle> { |
There was a problem hiding this comment.
Out of curiosity, is it possible to use File::duplicate from libstd?
There was a problem hiding this comment.
Hmm, good question. I bet it would be as, as far as I can remember, File::duplicate at the very bottom uses DuplicateHandle syscall.
There was a problem hiding this comment.
Yeah I was reviewing and the code was pretty similar to libstd here, so was just a passing though. In general though avoiding platform-specific code where possible and using libstd aggressively would also naturally fix things like this.
| vec: ws2def::WSABUF, | ||
| _p: PhantomData<&'a mut [u8]>, | ||
| pub unsafe fn ciovec_to_win<'a>(ciovec: &'a host::__wasi_ciovec_t) -> winx::io::IoVec<'a> { | ||
| let slice = slice::from_raw_parts(ciovec.buf as *const u8, ciovec.buf_len); |
There was a problem hiding this comment.
This is sort of a random drive-by comment, but this seems like it might be a useful API to share across Windows/Unix?
There was a problem hiding this comment.
That's an excellent observation which becomes even more relevant given your comment below about vectored IO getting stabilised really soon :-) I reckon then we could indeed nicely consolidate ciovec for both platforms by using IoSlice and IoSliceMut as the return type.
src/sys/windows/hostcalls/fs.rs
Outdated
| &mut host_nwritten, | ||
| std::ptr::null_mut(), | ||
| ) | ||
| let host_nwritten = match winx::io::writev(fe.fd_object.raw_handle, &iovs) { |
There was a problem hiding this comment.
Although not stable yet (but next week will be!) it'd be cool to get an underlying &File for both Unix and Windows and call write_vectored here, and that way there wouldn't need to be any platform-specific code at all for handling Unix/Windows and writev for files.
It does get a little more complicated with sockets, but it's more complicated anyway with sockets in general. It may also be solvable by having a platform-agnostic API to get something like an io::Write or io::Read from a file descriptor number for operations like this.
There was a problem hiding this comment.
Oh nice one! You've probably already noticed that I've based winx::io::IoVec on IoSlice from libstd so I'm really excited for write_vectored becoming stable! I'll for sure revisit redoing the writev/readv functionality with write_/read_vectored!
In terms of sockets, you can say that again! I've been essentially trying to steer away from any socket considerations for now, but if libstd would handle it transparently behind the scene, that would be magic! In fact, if you feel I could help out with anything in libstd, just give me a shout! I've been scrutinising it quite a bit over the past month or so, and would definitely be happy to lend a hand wherever I can!
|
@alexcrichton First off, I'm a big fan of your work! ;-)
By all means, please do! In fact, you could even make it a habit as the more comments the better the code in the end!
Heh, funny that you mention that as I've in fact been basing the vast majority of the code on either |
|
Reviewing your comments, I think they're spot on, and I'd really appreciate it if you could do it more often, whenever you can spare some of your time! Also, feel free to simply reach out to me offline with some suggestions and I'd be more than happy to discuss and act on them! Now, addressing your current comments that you've left so far, I'll be really happy to investigate them in this PR even if it means postponing merging it somewhat, but first I'd like to get #29 merged into master and then backported into this branch, so that the integration tests are in place :-) |
|
Heh thanks for the kind words as well :) I don't mean to block this by any means, it looks like it fits well enough into the current architecture and is probably more worthwhile to iterate in-tree rather than attempting to get a perfect PR. I'm not personally a huge fan of using crates like |
You're not blocking anything. A good code review is always welcome!
@alexcrichton first of, thanks for your input! It's really useful! I might repeat myself here but keep them coming! Now, I see two routes we can take here: 1) we leave this PR as-is and as @alexcrichton pointed out, and in subsequent PRs substitute as much host-specific calls with |
|
I agree with keeping this PR as-is and iterating on stuff in-tree. |
|
@sunfishcode @alexcrichton in-tree it is then! @alexcrichton I hope you won't mind if I make you a reviewer from time to time when re-using as much |
|
Yeah feel free to cc me on PRs for things like this, I may not have time to get to them immediately but I can certainly try to make sure I've got time to give feedback! |
|
Ok, let's merge this and continue to iterate in-tree! |
The aim of this PR is as in the title: to implement enough Windows hostcalls in order to successfully run our WASI tutorial. It builds directly upon #21.
I'll post any updates regarding it here.
TODO:
fd_fdstat_getpath_open(EDIT: doesn't handle symlinks yet)fd_readfd_close