At the moment, when vfkit connects to gvisor-tap-vsock, it sends a "handshake" packet that gvisor-tap-vsock uses to know the network address it should send the packets to. In theory, we should be able to read the address of the first packet we get using MSG_PEEK, but I did not manage to get this to work at the time.
Discussions in containers/libkrun#367 (comment) make me think we could revisit it, and that it should work.