my takeaway would be:
  - there should be no special behavior for localhost
  - always perform port and userinfo validation iff respective part is present in domain field of SIWE message. If not, only check the domain parts supplied in the domain field.

It is the URI scheme that defines the default port value, so it's necessary for the interpretation of the domain that doesn't contain the :port.
eg. If we assume http as the URI scheme, lack of :port can be interpreted to mean :80 or empty. the RFC-3986 itself is not clear about it and we'd have to reach for the http spec, where (I assume from prior experience) the interpretation ends up being :80 not empty.

ERC-4361 would have to define domain as RFC-3986 authority with a URI scheme that doesn't specify a default port to allow ignoring the port under all circumstances.

I think the lack of a field defining the URI scheme in ERC-4361 spec may be considered as it being implicitly declared void. The domain is coming from a http or https scheme though.

...

Unless ERC-4361 is updated to explicitly define URI scheme for authority used as domain, it's on us to decide how to interpret the missing port.