Skip to content
This repository was archived by the owner on Nov 9, 2019. It is now read-only.

Add enough Windows functionality to run WASI tutorial#22

Merged
sunfishcode merged 23 commits intoCraneStation:masterfrom
kubkon:win-wasi-tutorial
Jun 28, 2019
Merged

Add enough Windows functionality to run WASI tutorial#22
sunfishcode merged 23 commits intoCraneStation:masterfrom
kubkon:win-wasi-tutorial

Conversation

@kubkon
Copy link
Copy Markdown
Member

@kubkon kubkon commented May 22, 2019

The aim of this PR is as in the title: to implement enough Windows hostcalls in order to successfully run our WASI tutorial. It builds directly upon #21.

I'll post any updates regarding it here.

TODO:

  • implement the basics of determine_type_rights on Windows
  • implement fd_fdstat_get
  • implement path_open (EDIT: doesn't handle symlinks yet)
  • implement fd_read
  • implement fd_close
  • implement error mapping Win => WASI (EDIT: partially complete)
  • WASI fdflags to Win? (this might be optional for the first PoC)
  • some Win tests would be great! (EDIT: this was addressed, at least preliminarily, in Integrate misc-tests into wasi-common #29)
  • refactor out common functionality between *nix and Win (EDIT: this was addressed in Move common functionality into hostcalls mod #26)

Copy link
Copy Markdown
Member

@sunfishcode sunfishcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far! Here are a few drive-by comments -- I realize this code is work-in-progress, so if these aren't helpful right now, feel free to disregard :-).

len,
&mut host_nwritten,
std::ptr::null_mut(),
)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this check the return value?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that definitely needs to be handled. In fact, I've just now done some work in that direction. I've decided to move some of the unsafe logic into a separate crate, winx. My plan with this crate is to essentially imitate nix crate (to a degree of course).

Anyhow, now the calls to close a raw Win handle check for errors, and so does duplicate and writing the IoVec to a handle. Anyhow, for the latter, have a look here.

let buf = iovecs
.iter()
.find(|b| !b.as_slice().is_empty())
.map_or(&[][..], |b| b.as_slice());
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if WSASend works on files? If not, should we iterate over the iovecs and call WriteFile on each individually?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an excellent question. I've been wondering about that myself as well. The short answer is, I'm not sure. However, the current implementation you've highlighted here is taken verbatim from Rust nightly's implementation. In Rust, a call to IoSlice write_vectored eventually resolves into default_write_vectored.

In Rust, IoSlice is also a wrapper around WSABUF, however, it seems it's reserved only for file operations which I'm guessing would exclude sockets? Anyhow, I think it might be prudent to have a closer look at WSASend as well as the general implementation advocated by Rust and see what's what, and what to use. For the moment, I'll leave it as is, however, I'll come back to it when I get the WASI tutorial working. Having said that though, if you come up with something, then please fire away!

| host::__WASI_RIGHT_FD_WRITE
| host::__WASI_RIGHT_FD_ALLOCATE
| host::__WASI_RIGHT_FD_FILESTAT_SET_SIZE)
!= 0;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the logic for determining if we need read and/or write here be factored into a utility function that can be shared with the unix impl?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, a great question! With the winx crate, I think the hostcalls will start to share more and more code between *nix and Windows, so we'll definitely have some room for refactoring here. Perhaps we could twist it slightly and have the hostcalls (as the public API entry) outside of host specific modules, and have them call host specific bits only? I'm wondering how messy would that be? Anyhow, definitely something on my radar! :-)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sharing code to this degree would be great, though not if it makes things more complicated, so we'll see where it goes :-). Factoring out the code for inferring read/write here would be especially nice because deciding exactly which rights determine this is tricky, and fairly likely to change as WASI evolves.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've factored out common functionality in #26. Have a look!

@kubkon
Copy link
Copy Markdown
Member Author

kubkon commented May 25, 2019

Looks good so far! Here are a few drive-by comments -- I realize this code is work-in-progress, so if these aren't helpful right now, feel free to disregard :-).

Oh no, please keep them coming! The more discussion we do early on, the better the output will be -- I'm a firm believer in that :-)

Copy link
Copy Markdown
Collaborator

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tschneidereit pointed me at this and I'm personally quite intrigued by all this so tried to leave some comments here and there!

One thing I was thinking is that it might be a good idea to reuse the standard library as much as possible. Within libstd (and throughout the Rust ecosystem) there's lots of trickiness in getting syscalls and such right, so it'd be great to be able to reuse all the the existing knowledge! Now this is more of a long-term thing of course in the sense that there's no need to get this perfect on the first try, but just something I was thinking!

// disk file: file, dir or disk device
let file = File::from_raw_handle(raw_handle);
let meta = file.metadata();
std::mem::forget(file);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW a helpful pattern I've found in the past is that instead of from_raw_* plus forget you can do:

let file = ManuallyDrop::new(File::from_raw_handle(raw_handle));
let meta = file.metadata();
// ...

and the usage of ManuallyDrop means you don't have to call forget and can continue to use the File if necessary

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one, thanks! That's a fantastic suggestion :-)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! For the moment, I've decided to only include it in the bits pertaining this very PR (so Win only), whereas I'll re-apply the suggestion to *nix bits in a separate PR.

}
}
winx::handle::close(self.raw_handle)
.unwrap_or_else(|e| eprintln!("FdObject::drop(): {}", e))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW we explicitly ignore errors in the Rust standard library for errors on close. We largely do that to mirror weirdness on Unix, but perhaps this should mirror that?

(or would it be possible to use a File from the standard library maybe?)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pointer! So the reason we've got error handling here is essentially because I mirrored the implementation here from the corresponding *nix implementation in wasi-common. Having said that, it might indeed be obsolete :-)

(Hmm, perhaps we could use File instead. If we could, then, again, we could consolidate a fair amount of *nix and Win code, so it'd be amazing!)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah that makes sense (mirroring) and I don't want to try to pile on too much into one PR which is just trying to get things done!

use std::os::windows::prelude::RawHandle;
use winapi::shared::minwindef::FALSE;

pub fn dup(old_handle: RawHandle) -> Result<RawHandle> {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, is it possible to use File::duplicate from libstd?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, good question. I bet it would be as, as far as I can remember, File::duplicate at the very bottom uses DuplicateHandle syscall.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was reviewing and the code was pretty similar to libstd here, so was just a passing though. In general though avoiding platform-specific code where possible and using libstd aggressively would also naturally fix things like this.

vec: ws2def::WSABUF,
_p: PhantomData<&'a mut [u8]>,
pub unsafe fn ciovec_to_win<'a>(ciovec: &'a host::__wasi_ciovec_t) -> winx::io::IoVec<'a> {
let slice = slice::from_raw_parts(ciovec.buf as *const u8, ciovec.buf_len);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sort of a random drive-by comment, but this seems like it might be a useful API to share across Windows/Unix?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an excellent observation which becomes even more relevant given your comment below about vectored IO getting stabilised really soon :-) I reckon then we could indeed nicely consolidate ciovec for both platforms by using IoSlice and IoSliceMut as the return type.

&mut host_nwritten,
std::ptr::null_mut(),
)
let host_nwritten = match winx::io::writev(fe.fd_object.raw_handle, &iovs) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although not stable yet (but next week will be!) it'd be cool to get an underlying &File for both Unix and Windows and call write_vectored here, and that way there wouldn't need to be any platform-specific code at all for handling Unix/Windows and writev for files.

It does get a little more complicated with sockets, but it's more complicated anyway with sockets in general. It may also be solvable by having a platform-agnostic API to get something like an io::Write or io::Read from a file descriptor number for operations like this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice one! You've probably already noticed that I've based winx::io::IoVec on IoSlice from libstd so I'm really excited for write_vectored becoming stable! I'll for sure revisit redoing the writev/readv functionality with write_/read_vectored!

In terms of sockets, you can say that again! I've been essentially trying to steer away from any socket considerations for now, but if libstd would handle it transparently behind the scene, that would be magic! In fact, if you feel I could help out with anything in libstd, just give me a shout! I've been scrutinising it quite a bit over the past month or so, and would definitely be happy to lend a hand wherever I can!

@kubkon
Copy link
Copy Markdown
Member Author

kubkon commented Jun 25, 2019

@alexcrichton First off, I'm a big fan of your work! ;-)

@tschneidereit pointed me at this and I'm personally quite intrigued by all this so tried to leave some comments here and there!

By all means, please do! In fact, you could even make it a habit as the more comments the better the code in the end!

One thing I was thinking is that it might be a good idea to reuse the standard library as much as possible. Within libstd (and throughout the Rust ecosystem) there's lots of trickiness in getting syscalls and such right, so it'd be great to be able to reuse all the the existing knowledge! Now this is more of a long-term thing of course in the sense that there's no need to get this perfect on the first try, but just something I was thinking!

Heh, funny that you mention that as I've in fact been basing the vast majority of the code on either libstd or the nix crate ;-) Anyhow, I totally agree that we should reuse and consolidate knowledge as much as possible, especially since it does prove to be quite tricky to get the syscalls right. I'm open to any and all pointers and suggestions!

@kubkon
Copy link
Copy Markdown
Member Author

kubkon commented Jun 25, 2019

Reviewing your comments, I think they're spot on, and I'd really appreciate it if you could do it more often, whenever you can spare some of your time! Also, feel free to simply reach out to me offline with some suggestions and I'd be more than happy to discuss and act on them!

Now, addressing your current comments that you've left so far, I'll be really happy to investigate them in this PR even if it means postponing merging it somewhat, but first I'd like to get #29 merged into master and then backported into this branch, so that the integration tests are in place :-)

@alexcrichton
Copy link
Copy Markdown
Collaborator

Heh thanks for the kind words as well :)

I don't mean to block this by any means, it looks like it fits well enough into the current architecture and is probably more worthwhile to iterate in-tree rather than attempting to get a perfect PR.

I'm not personally a huge fan of using crates like nix when writing a portability layer like this since it brings in its own level of complexity and leaves out a major platform (Windows). I find myself inevitably ending up using the raw libc bindings anyway, but that may just be me! In any case trying to use File in libstd or eventually things like TcpStream I think will also help quite a lot here in that almost all of these operations are either there or core crates on crates.io, and using those would reduce the amount of platform-specific code that has to be here in theory

@kubkon
Copy link
Copy Markdown
Member Author

kubkon commented Jun 26, 2019

Heh thanks for the kind words as well :)

I don't mean to block this by any means, it looks like it fits well enough into the current architecture and is probably more worthwhile to iterate in-tree rather than attempting to get a perfect PR.

You're not blocking anything. A good code review is always welcome!

I'm not personally a huge fan of using crates like nix when writing a portability layer like this since it brings in its own level of complexity and leaves out a major platform (Windows). I find myself inevitably ending up using the raw libc bindings anyway, but that may just be me! In any case trying to use File in libstd or eventually things like TcpStream I think will also help quite a lot here in that almost all of these operations are either there or core crates on crates.io, and using those would reduce the amount of platform-specific code that has to be here in theory

@alexcrichton first of, thanks for your input! It's really useful! I might repeat myself here but keep them coming!

Now, I see two routes we can take here: 1) we leave this PR as-is and as @alexcrichton pointed out, and in subsequent PRs substitute as much host-specific calls with libstd as possible, or 2) we try to re-use libstd as much as possible in this PR already and see where it takes us at the expense of delaying this PR somewhat. @sunfishcode let me know what you reckon. I already know that @alexcrichton, although in favour of libstd, would rather not turn this PR upside-down and iterate in-tree, but I'm curious to see what you reckon. Personally, I'm OK with either approach :-)

@sunfishcode
Copy link
Copy Markdown
Member

I agree with keeping this PR as-is and iterating on stuff in-tree.

@kubkon
Copy link
Copy Markdown
Member Author

kubkon commented Jun 27, 2019

@sunfishcode @alexcrichton in-tree it is then!

@alexcrichton I hope you won't mind if I make you a reviewer from time to time when re-using as much libstd as possible in the near future in the upcoming PRs?

@alexcrichton
Copy link
Copy Markdown
Collaborator

Yeah feel free to cc me on PRs for things like this, I may not have time to get to them immediately but I can certainly try to make sure I've got time to give feedback!

@kubkon kubkon marked this pull request as ready for review June 27, 2019 17:01
@sunfishcode
Copy link
Copy Markdown
Member

Ok, let's merge this and continue to iterate in-tree!

@sunfishcode sunfishcode merged commit 7287767 into CraneStation:master Jun 28, 2019
@kubkon kubkon deleted the win-wasi-tutorial branch June 29, 2019 08:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants