What should be returned when you read the contents of a directory? This PR only returns the names of the contents, libuv does expose the kind of the entries which probably would be nice to add?

Yes, getdents on Linux gives you the kind, which saves having to stat each entry in some cases. Though it can also return DT_UNKNOWN if the FS doesn't support it.

At the moment I don't see a way to provide an asynchronous version with uring?

We can use systhreads for things that uring doesn't handle. We'll probably need to do that anyway for stuff like gethostbyname.

1. Why is it read_dir : Eio.Dir.t -> path -> string list? I expected read_dir : Eio.Dir.t -> string list - "given Eio.Dir.t give me its contents"
2. You should consider how the API will be used. If you give me a list of strings, I will usually have to check each one to see whether it is a directory or a file because the API for working with directories and files is different. How about a list of the following?

   type dirent = [ 
   | `File of path
   | `Dir of Eio.Dir.t
   ]

3. What if a directory contains 10**12 files (see) and I don't want a list that big in memory? There should be an API to read the contents of the directory asynchronously just like we are able to read a file asynchronously

Thanks both for the comments :))

Though it can also return DT_UNKNOWN if the FS doesn't support it.

Yep, this is also exposed in luv https://aantron.github.io/luv/luv/Luv/File/Dirent/Kind/index.html#type-t.UNKNOWN

We can use systhreads for things that uring doesn't handle. We'll probably need to do that anyway for stuff like gethostbyname.

Forgot this was an option, I'll have a go!

Why is it read_dir : Eio.Dir.t -> path -> string list? I expected read_dir : Eio.Dir.t -> string list - "given Eio.Dir.t give me its contents"

My understanding is that an Eio.Dir.t is not a directory, but instead a file-system access capability. Maybe Dir is a bit misleading, perhaps Eio.Fs is better, but maybe @talex5 has good reason for the naming scheme? In the case of cwd it ensures we can't access things like ../../some/dir whereas the full-system access capability fs would allow such a thing but note that it doesn't actually impact the path, it just checks what we're trying to access with the path.

There's also https://github.com/ocaml-multicore/eio#design-note-capabilities for some more context on this design.

How about a list of the following?

type dirent = [ 
| `File of path
| `Dir of Eio.Dir.t
]

Yep returning the kinds makes sense. Maybe following the layout of luv so:

type dirent = {
  kind : kind;
  name : string;
}
and kind = [ `Unknown | `Dir | `File ... ]

`Dir of Eio.Dir.t isn't quite right given the explanation of Eio.Dir.t above.

What if a directory contains 10**12 files (see) and I don't want a list that big in memory?

Great point, I'll see what's available in the backend APIs to make this possible :))

Given:

let cwd = Eio.Stdenv.cwd env in
Eio.Dir.read_dir cwd "path/to/foo"

will the result be path/to/foo/bar.txt or bar.txt? I think path/to/foo/bar.txt would be more convenient.

Currently it would bebar.txt which I think is in line with readdir(3), Sys.readdir and Luv.File.readdir so if we want paths to the directory entries, it probably shouldn't be called read_dir. Reconstructing the path (once we have a better path type) wouldn't be too hard and maybe we could provide a convenience function built on top of read_dir for this? Alternatively, I suppose, it won't be too hard to provide a Path.filename function too.

Why is it read_dir : Eio.Dir.t -> path -> string list? I expected read_dir : Eio.Dir.t -> string list - "given Eio.Dir.t give me its contents"

My understanding is that an Eio.Dir.t is not a directory, but instead a file-system access capability. Maybe Dir is a bit misleading, perhaps Eio.Fs is better, but maybe @talex5 has good reason for the naming scheme?

I just copied Rust's name for it (https://docs.rs/cap-std/0.22.1/cap_std/fs/struct.Dir.html). They have e.g.

impl Dir {
  fn read_dir<P: AsRef<Path>>(&self, path: P) -> Result<ReadDir>

The Eio API also matches the POSIX one (e.g. openat2, where you pass a base FD and a path relative to that). The reason for having both is mostly to do with symlinks: you can't access the parent of the base FD, but you can follow symlinks within the subtree.

So the idea is that a Dir.t is a directory, and doesn't allow access to its parent. However, fs is an exception to this rule because it represents the current directory but also allows access to everything (via .., symlinks, or an absolute path).

I pushed a commit which tries to use systhreads now they're fixed in OCaml trunk. It's meant to generate a discussion about how best to use them to provide non-blocking, asynchronous versions of unsupported functions in Uring.

Right now the implementation uses a new effect, Unblock which takes a function to "unblock". It runs this function in a newly created systhread and the result is then enqueued to the run queue. I don't know if this is the general idea that you had in mind @talex5 ? I had a look at Lwt and Mwt which are more complicated with pools of threads and workers, but I thought thanks to effects we might be able to simplify that a fair bit.

Also the readdir function reuses openat2 to do the necessary filesystem containment checking, but then opens a directory handle anyway which probably isn't particularly efficient.

Yes, probably makes sense for the backend API to provide open_dir, etc, and for read_dir to be a wrapper around that.

Should Eio.Dir.read_dir try to read all directory entries or just some amount of them and the user can call it multiple times ? Or should that be a completely separate function ?