ArcadeDB Server v26.7.1-SNAPSHOT (build 661e52fce3b84f452f02cc131b9c5608674deb67/1782283285125/main)
In our HA setup with 3 nodes, the serving of a database snaphot fails with a timeout.
Log from the follower
java[72080]: 2026-06-24 10:48:34.157 INFO [ArcadeStateMachine] Database 'graph-data' bootstrap mismatch (local lastTxId=109387 / fp=23b2cb30..., baseline lastTxId=112538 / fp=37cf39b7...); reinstalling from leader-shipped full snapshot
java[72080]: 2026-06-24 10:53:34.215 WARNI [SnapshotInstaller] Snapshot download attempt 1/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 10:58:39.770 WARNI [SnapshotInstaller] Snapshot download attempt 2/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:03:50.376 WARNI [SnapshotInstaller] Snapshot download attempt 3/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:09:10.978 WARNI [SnapshotInstaller] Snapshot download attempt 4/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:14:11.596 SEVER [ArcadeStateMachine] Failed to install snapshot during bootstrap for database 'graph-data': Failed to install snapshot for bootstrap-mismatched database 'graph-data'. Keeping the local copy and scheduling an async retry once a leader is reachable.
java[72080]: 2026-06-24 11:19:17.238 WARNI [SnapshotInstaller] Snapshot download attempt 1/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:24:27.865 WARNI [SnapshotInstaller] Snapshot download attempt 2/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:24:27.865 WARNI [SnapshotInstaller] Snapshot download attempt 3/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:29:48.493 WARNI [SnapshotInstaller] Snapshot download attempt 4/4 failed for 'graph-data': Premature EOF
java[72080]: 2026-06-24 11:29:49.181 SEVER [ArcadeStateMachine] Snapshot download triggered by watchdog failed
java[72080]: java.io.IOException: Snapshot download failed after 4 attempts for 'graph-data'
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:522)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:185)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:137)
java[72080]: at com.arcadedb.server.ha.raft.ArcadeStateMachine.triggerSnapshotDownload(ArcadeStateMachine.java:1265)
java[72080]: at com.arcadedb.server.ha.raft.ArcadeStateMachine.lambda$applyBootstrapFingerprintEntry$6(ArcadeStateMachine.java:1076)
java[72080]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
java[72080]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
java[72080]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java[72080]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java[72080]: at java.base/java.lang.Thread.run(Thread.java:1583)
java[72080]: Caused by: java.io.IOException: Premature EOF
java[72080]: at java.base/sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:567)
java[72080]: at java.base/sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:611)
java[72080]: at java.base/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:705)
java[72080]: at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
java[72080]: at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3707)
java[72080]: at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller$CountingInputStream.read(SnapshotInstaller.java:689)
java[72080]: at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
java[72080]: at java.base/java.io.PushbackInputStream.read(PushbackInputStream.java:197)
java[72080]: at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:264)
java[72080]: at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175)
java[72080]: at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:412)
java[72080]: at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.copyWithLimit(SnapshotInstaller.java:650)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadSnapshot(SnapshotInstaller.java:586)
java[72080]: at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:512)
java[72080]: ... 9 more
Log from the leader
java[115047]: 2026-06-24 11:26:20.599 INFO [SnapshotHttpHandler] Serving database snapshot for 'graph-data'...
java[115047]: 2026-06-24 11:29:48.489 WARNI [SnapshotHttpHandler] Snapshot write for 'graph-data' timed out after 300000ms, closing connection to release semaphore slot
java[115047]: 2026-06-24 11:29:48.491 SEVER [SnapshotHttpHandler] Error serving snapshot for 'graph-data'
java[115047]: java.io.IOException: UT000034: Stream is closed
java[115047]: at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:232)
java[115047]: at java.base/java.util.zip.ZipOutputStream.write(ZipOutputStream.java:361)
java[115047]: at java.base/java.io.InputStream.transferTo(InputStream.java:797)
java[115047]: at java.base/java.io.FileInputStream.transferTo(FileInputStream.java:390)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.addFileToZip(SnapshotHttpHandler.java:318)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:283)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$2(SnapshotHttpHandler.java:169)
java[115047]: at com.arcadedb.utility.CodeUtils.executeIgnoringExceptions(CodeUtils.java:39)
java[115047]: at com.arcadedb.engine.PageManager.suspendFlushAndExecute(PageManager.java:147)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$3(SnapshotHttpHandler.java:167)
java[115047]: at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1730)
java[115047]: at com.arcadedb.server.ha.raft.RaftReplicatedDatabase.executeInReadLock(RaftReplicatedDatabase.java:1023)
java[115047]: at com.arcadedb.server.ServerDatabase.executeInReadLock(ServerDatabase.java:597)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.handleRequest(SnapshotHttpHandler.java:161)
java[115047]: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:368)
java[115047]: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:935)
java[115047]: at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
java[115047]: at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
java[115047]: at java.base/java.lang.Thread.run(Thread.java:1583)
java[115047]: Suppressed: java.io.IOException: UT000034: Stream is closed
java[115047]: at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]: at java.base/java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:283)
java[115047]: at java.base/java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:388)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:267)
java[115047]: at java.base/java.util.zip.ZipOutputStream.close(ZipOutputStream.java:405)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:269)
java[115047]: ... 17 more
java[115047]: 2026-06-24 11:29:48.491 SEVER [CodeUtils] Error during suspend flush
java[115047]: java.lang.RuntimeException: java.io.IOException: UT000034: Stream is closed
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:300)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$2(SnapshotHttpHandler.java:169)
java[115047]: at com.arcadedb.utility.CodeUtils.executeIgnoringExceptions(CodeUtils.java:39)
java[115047]: at com.arcadedb.engine.PageManager.suspendFlushAndExecute(PageManager.java:147)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$3(SnapshotHttpHandler.java:167)
java[115047]: at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1730)
java[115047]: at com.arcadedb.server.ha.raft.RaftReplicatedDatabase.executeInReadLock(RaftReplicatedDatabase.java:1023)
java[115047]: at com.arcadedb.server.ServerDatabase.executeInReadLock(ServerDatabase.java:597)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.handleRequest(SnapshotHttpHandler.java:161)
java[115047]: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:368)
java[115047]: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:935)
java[115047]: at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
java[115047]: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
java[115047]: at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
java[115047]: at java.base/java.lang.Thread.run(Thread.java:1583)
java[115047]: Caused by: java.io.IOException: UT000034: Stream is closed
java[115047]: at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:232)
java[115047]: at java.base/java.util.zip.ZipOutputStream.write(ZipOutputStream.java:361)
java[115047]: at java.base/java.io.InputStream.transferTo(InputStream.java:797)
java[115047]: at java.base/java.io.FileInputStream.transferTo(FileInputStream.java:390)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.addFileToZip(SnapshotHttpHandler.java:318)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:283)
java[115047]: ... 17 more
java[115047]: Suppressed: java.io.IOException: UT000034: Stream is closed
java[115047]: at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]: at java.base/java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:283)
java[115047]: at java.base/java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:388)
java[115047]: at java.base/java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:267)
java[115047]: at java.base/java.util.zip.ZipOutputStream.close(ZipOutputStream.java:405)
java[115047]: at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:269)
java[115047]: ... 17 more
ArcadeDB Server v26.7.1-SNAPSHOT (build 661e52fce3b84f452f02cc131b9c5608674deb67/1782283285125/main)In our HA setup with 3 nodes, the serving of a database snaphot fails with a timeout.
Log from the follower
Log from the leader