This surfaced when fixing the 0rtt flakes in #3496
Originally the PR also invalidated old paths for the same IP when a new one is encountered, which happened if you restarted the server and tried to re-connect.

However that turned out to be a bit risky as we don't have a complete understanding of possible side effects so that part of the PR was reverted. Keeping this issue up so we can follow up.

Reference code 110fb33