Goal

Make the errors for certificate and host name errors clearer.

Why

CERT_UNTRUSTED is a very unhelpful error, because it has no information about why the cert was untrusted. This makes debugging issue very irritating.

This PR makes two changes

• hostname verification now has it's own error, clearly telling you that certificate was for the wrong entity and that's why it's not trusted.
• "security policy" failures have their own error. This is useful because it tells an endpoint owner that the cert wasn't rejected for e.g. an incorrect signature, but rather because the endpoint owner set a policy preventing the cert from being used.

How

We add a new error for the host name verification issue. We adopt the existing S2N_ERR_SECURITY_POLICY_INCOMPATIBLE_CERT issue for the mTLS failure case.

Callouts

I don't love that S2N_ERR_SECURITY_POLICY_INCOMPATIBLE_CERT is a UsageError. However, it's also not totally correct to call it a ProtocolError.

Honestly, I think that UsageError is more accurate than ProtocolError, and I think the benefit of the much clearer error outweighs any downsides to the slightly odd Error Type.

Testing

I am relying on the integration tests I added in #5755

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.