I had a look at lwt_luv to see how they handle it:

open Lwt.Infix

let () =
  let () = Lwt_engine.set (new Lwt_luv.engine) in
  let socket, _ = Lwt_unix.(socketpair PF_UNIX SOCK_STREAM 0) in
  Lwt_main.run begin
    let _a = Lwt_unix.wait_read socket in
    let rec aux () =
      Lwt_unix.wait_write socket >>= fun () ->
      Lwt_unix.write socket (Bytes.of_string "a") 0 1 >>= fun (_ : int) ->
      aux ()
    in
    aux ()
  end

The result is:

Fatal error: exception Failure(
"Could not register fd for write polling, this is probably a error in Lwt,
please open a issue on the repo. \n
Error message: EEXIST")

However, Lwt does handle the case of multiple readers OK (just not a mix of readers and writers):
https://github.com/ocsigen/lwt/blob/5.6.1/src/unix/lwt_unix.cppo.ml#L495

I believe the logic for when to stop a handle is currently incorrect, a fiber could be cancelled but this doesn't remove it from the Lf_queue and I'm using the Lf_queue size to check whether there are any fibers waiting on the handle -- perhaps an integer that we decrement/increment would suffice here?

You probably want a Lwt_dllist here instead of an Lf_queue. There shouldn't be any need for anything lock-free here, since we run a separate luv event loop in each domain.

This means the same handle must register for both kinds of events and then we decided how to handle them

I don't think this will work. http://docs.libuv.org/en/v1.x/poll.html#c.uv_poll_start says:

If one of the events UV_READABLE or UV_WRITABLE are set, the callback will be called again, as long as the given fd/socket remains readable or writable accordingly.

However, it also seems unnecessary:

Note: Calling uv_poll_start() on a handle that is already active is fine. Doing so will update the events mask that is being watched for.

It's not clear to me what happens to the callback if you call start multiple times. Possibly it ignores the second one, or disconnects the first. In any case, stop doesn't say which callback to stop.

Yes, I'm not really sure what the actual behaviour is. From the source code, it would seem after calling Poll.start multiple times a new callback is installed each time https://github.com/libuv/libuv/blob/988f2bfc4defb9a85a536a3e645834c161143ee0/src/unix/poll.c#L152. In this tutorial under "Setting up Polling", it would seem to suggest this is the case and they're happy enough because it is the same callback installed each time.

In b12c0d6 I've made it such that callbacks are the same and a cancelled fiber will remove itself from the list of continuations to enqueue so the is_empty check (in my mind) in safe_to_stop_polling is now correct.

In 4999630 I've added the extra logic to handle simultaneously awaiting for writability and readability on a FD. The bit mask for events is updated to whatever you pass to Poll.start, I had thought it might bitwise OR with whatever is already there, this is not the case.

Yes your changes are very sensible :)) What tool do you use for indentation and whitespaces, ocp-indent?

Do we need to fail all waiters if close is called too?

This seems reasonable, the docs say:

The user should not close a file descriptor while it is being polled by an active poll handle. This can cause the handle to report an error, but it might also start polling another socket. However the fd can be safely closed immediately after a call to uv_poll_stop() or uv_close().

It seems reasonable to me to fail any suspended fibers, would this approach work on Linux?

One removes the use of Obj.magic ...

Hmm, this doesn't work for all handles. I guess the FD isn't known initially in some cases.

OK, fixed it by adding a constraint. It seems we only ever use handles types that may have FDs anyway. I also got it to ignore EBADF when getting the FD, since sockets don't get their FD set immediately (e.g. a failure after TCP.init and before bind wouldn't have any FD).

OK, fixed it by adding a constraint. It seems we only ever use handles types that may have FDs anyway.

Ah, I wasn't sure if that was the case. Definitely better than Obj.magic ! All looks good to me :))

Thanks!