The IOLoop isn't supposed to call handlers that have been removed, so you shouldn't be reaching this point. The default IOLoop removes any pending callbacks from self._events, and while I don't think AsyncIOLoop handles this as gracefully it doesn't look like it could call into a removed callback.

That means that the sequence of events is not "queue accept handler, stop server, call accept handler", it's "call accept handler, which synchronously stops the server while add_accept_handler's callback is still on the stack". (this happens sporadically depending on whether the packets arrive fast enough for all the recv calls to be satisfied on the fast synchronous path)

This could have a simpler solution: have add_accept_handler schedule the callbacks with IOLoop.current().add_callback instead of running them directly. This would ensure that you can't close the listening socket while the accept handler is running. However, it would be a slight behavior change in multi-process mode that could result in worse load balancing (since one process will be able to pull multiple connections off the backlog faster. This is less of a concern with SO_REUSEPORT).