I have setup a 3 node cluster on Kube with SSL enabled on port 2490 ...
-Darcadedb.ssl.enabled=true
# the keystore and truststores are setup
-Darcadedb.ssl.keyStore=..."
-Darcadedb.ssl.keyStorePassword=..."
-Darcadedb.ssl.trustStore=..."
-Darcadedb.ssl.trustStorePassword=..."
And client connections to insert data (using curl --cacert ....) are functional via https URL. 👍
A default database is created via server startup args like -Darcadedb.server.defaultDatabases=dbinst1[root]
When the cluster is up, I am creating a new database school ...and that gets created 👍
curl -k -X POST https://<arcade-svc>.<namespace>.svc.cluster.local:2490/api/v1/server \
-d '{"command": "create database school"}' \
-H "Content-Type: application/json" \
--user root:${ROOT_PASSWORD}
After some time I notice an issue in the logs which is causing the server container to restart in the pod.
Here is the log...
2026-06-01 15:36:47.189 INFO [ArcadeStateMachine] Database 'school' bootstrap mismatch (local lastTxId=0 / fp=442c8156..., baseline lastTxId=4 / fp=442c8156...); reinstalling from leader-shipped full snapshot
2026-06-01 15:36:47.308 WARNI [SnapshotInstaller] Snapshot download attempt 1/4 failed for 'school': Unsupported or unrecognized SSL message
2026-06-01 15:36:52.339 WARNI [SnapshotInstaller] Snapshot download attempt 2/4 failed for 'school': Unsupported or unrecognized SSL message
2026-06-01 15:37:02.371 WARNI [SnapshotInstaller] Snapshot download attempt 3/4 failed for 'school': Unsupported or unrecognized SSL message
2026-06-01 15:37:22.398 WARNI [SnapshotInstaller] Snapshot download attempt 4/4 failed for 'school': Unsupported or unrecognized SSL message
2026-06-01 15:37:22.399 SEVER [ArcadeStateMachine] CRITICAL: Unexpected error applying Raft log entry at index 1. Shutting down to prevent state divergence.
java.lang.RuntimeException: Failed to install snapshot for bootstrap-mismatched database 'school'
at com.arcadedb.server.ha.raft.ArcadeStateMachine.installFromLeaderForBootstrap(ArcadeStateMachine.java:894)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.applyBootstrapFingerprintEntry(ArcadeStateMachine.java:863)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.applyTransaction(ArcadeStateMachine.java:260)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1888)
at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:255)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:194)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: Snapshot download failed after 4 attempts for 'school'
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:281)
at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:129)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.installFromLeaderForBootstrap(ArcadeStateMachine.java:890)
... 6 more
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:462)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:586)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1705)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1629)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadSnapshot(SnapshotInstaller.java:314)
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:271)
... 8 more
2026-06-01 15:37:22.404 INFO [ArcadeDBServer] Shutting down ArcadeDB Server...
2026-06-01 15:37:22.405 INFO [PluginManager] - Stop RedisProtocolPlugin plugin
2026-06-01 15:37:22.407 INFO [PluginManager] - Stop PostgresProtocolPlugin plugin
2026-06-01 15:37:22.408 INFO [PluginManager] - Stop PrometheusMetricsPlugin plugin
2026-06-01 15:37:22.409 INFO [PluginManager] - Stop RaftHAPlugin plugin
2026-06-01 15:37:22.417 SEVER [StateMachineUpdater] Exception while wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434@group-E6A6271798FD-StateMachineUpdater: applying txn index=1, nextLog=(t:1, i:1), STATEMACHINELOGENTRY, WRITE: logData.size=85, stateMachineData.size=0, logEntryProtoSerializedSize=0
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to install snapshot for bootstrap-mismatched database 'school'
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:874)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357)
at org.apache.ratis.server.impl.RaftServerImpl.replyPendingRequest(RaftServerImpl.java:1816)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1889)
at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:255)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:194)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.RuntimeException: Failed to install snapshot for bootstrap-mismatched database 'school'
at com.arcadedb.server.ha.raft.ArcadeStateMachine.installFromLeaderForBootstrap(ArcadeStateMachine.java:894)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.applyBootstrapFingerprintEntry(ArcadeStateMachine.java:863)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.applyTransaction(ArcadeStateMachine.java:260)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1888)
... 3 more
Caused by: java.io.IOException: Snapshot download failed after 4 attempts for 'school'
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:281)
at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:129)
at com.arcadedb.server.ha.raft.ArcadeStateMachine.installFromLeaderForBootstrap(ArcadeStateMachine.java:890)
... 6 more
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:462)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:586)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1705)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1629)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307)
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadSnapshot(SnapshotInstaller.java:314)
at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:271)
... 8 more
2026-06-01 15:37:22.425 INFO [ArcadeStateMachine] Database 'dbinst1' bootstrapped locally (lastTxId=0, fingerprint matches cluster baseline)
2026-06-01 15:37:22.507 INFO [RaftClusterManager] Peer wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434 removed from Raft cluster
2026-06-01 15:37:23.472 INFO [RaftHAServer] RaftHAServer stopped
2026-06-01 15:37:23.475 INFO [undertow] stopping server: Undertow - 2.4.0.FinalReceived shutdown signal. The server will be halted
- Stop JMX Metrics
ArcadeDB Server is down
Looking closely at the source indicates a problem with creating the TrustManager for the client to download the snapshot. The trust manager for the client is being built with the keystore and not the truststore which is incorrect - https://github.com/ArcadeData/arcadedb/blob/main/ha-raft/src/main/java/com/arcadedb/server/ha/raft/SnapshotInstaller.java#L409
I have setup a 3 node cluster on Kube with SSL enabled on port 2490 ...
And client connections to insert data (using curl --cacert ....) are functional via https URL. 👍
A default database is created via server startup args like
-Darcadedb.server.defaultDatabases=dbinst1[root]When the cluster is up, I am creating a new database
school...and that gets created 👍After some time I notice an issue in the logs which is causing the server container to restart in the pod.
Here is the log...
Looking closely at the source indicates a problem with creating the TrustManager for the client to download the snapshot. The trust manager for the client is being built with the keystore and not the truststore which is incorrect - https://github.com/ArcadeData/arcadedb/blob/main/ha-raft/src/main/java/com/arcadedb/server/ha/raft/SnapshotInstaller.java#L409