In the release note for quic-go v0.52.0, it's stated that

QUIC listeners created by the HTTP/3 server (i.e. when using http3.Server.ListenAndServe and ListenAndServeTLS) are immediately closed (quic-go/quic-go#5101). QUIC listeners created by the application (i.e. when using http3.Server.ServeListener) are left running, it is the application's responsibility to close them, or use them in a new server instance (quic-go/quic-go#5129).

Right now, caddy assumes that http3 server will close these listeners as before. A fix is under way.