Description

Connecting to a private MQTT broker over TLS by IP address fails on Android with a confusing platform error:

TLS handshake failed: Domain specific configurations require that hostname aware checkServerTrusted(X509Certificate[], String, String) is used

Why: ktor's CIO TLS engine only calls the 2-arg X509TrustManager.checkServerTrusted(chain, authType) (KTOR-2243). On Android, when the consuming app's network_security_config.xml contains any <domain-config> block, the platform swaps in a NetworkSecurityTrustManager that throws from the 2-arg overload — regardless of the target host — and demands the 3-arg hostname-aware overload.

5591751 already addressed this with HostnameAwareTrustManager, but only installed it when a TLS SNI server name was present. SNI is intentionally null for IP literals (RFC 6066 §3 forbids IPs in SNI), so configurePlatformTrust returned early and ktor fell back to the throwing platform default. A private broker reached by LAN IP — the common self-signed-cert setup — therefore still failed.

The defect was surfaced by the recent migration off OkHttp (whose trust manager calls the 3-arg overload) to ktor CIO. Diagnosis credit to @Olli-LUT on the linked issue.

This fix decouples the SNI server name (correctly null for IPs) from the trust-evaluation host (always the real host). The hostname-aware trust manager is now installed for IP-literal brokers too, routing through X509TrustManagerExtensions so the platform's domain-aware requirement is satisfied.

Note: this does not auto-trust self-signed certificates. After this fix an untrusted self-signed broker fails with a meaningful Trust anchor for certification path not found instead of the misleading internal error; to connect, the broker's CA must be installed in Android's user trust store.

Changes

All in the :transport-tcp module:

• configurePlatformTrust(serverName: String?) → configurePlatformTrust(host: String) across the expect and all three actuals (android / jvm / native). The non-null String is a compile-time guard against re-passing the nullable SNI value.
• Android actual now installs HostnameAwareTrustManager for any non-blank host (IP literal or DNS name), not only when SNI is present.
• TcpTransport call site passes endpoint.host for trust evaluation while still suppressing SNI for IPs.
• Extracted sniServerName() / isIpLiteral() to top-level internal functions and added commonTest's TcpTransportTlsTest covering the SNI-vs-trust-host decoupling (IPv4/IPv6 suppress SNI; DNS keeps it).

Testing

• spotlessCheck + detekt pass
• :transport-tcp:allTests passes (JVM + macOS + iOS native), incl. new TcpTransportTlsTest (4/4)
• :transport-tcp:compileCommonMainKotlinMetadata + Android target compile/assemble pass (expect/actual contract validated for all targets)
• New/updated tests cover the changes

Related Issues

Addresses meshtastic/Meshtastic-Android#5894 (an app-side mqtt-client version bump is required to ship the fix to users).