Skip to content

HA snapshot serving timed out after 300000ms #4729

Description

@danieljuhl

ArcadeDB Server v26.7.1-SNAPSHOT (build 661e52fce3b84f452f02cc131b9c5608674deb67/1782283285125/main)

In our HA setup with 3 nodes, the serving of a database snaphot fails with a timeout.

Log from the follower

 java[72080]: 2026-06-24 10:48:34.157 INFO  [ArcadeStateMachine] Database 'graph-data' bootstrap mismatch (local lastTxId=109387 / fp=23b2cb30..., baseline lastTxId=112538 / fp=37cf39b7...); reinstalling from leader-shipped full snapshot
 java[72080]: 2026-06-24 10:53:34.215 WARNI [SnapshotInstaller] Snapshot download attempt 1/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 10:58:39.770 WARNI [SnapshotInstaller] Snapshot download attempt 2/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:03:50.376 WARNI [SnapshotInstaller] Snapshot download attempt 3/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:09:10.978 WARNI [SnapshotInstaller] Snapshot download attempt 4/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:14:11.596 SEVER [ArcadeStateMachine] Failed to install snapshot during bootstrap for database 'graph-data': Failed to install snapshot for bootstrap-mismatched database 'graph-data'. Keeping the local copy and scheduling an async retry once a leader is reachable.
 java[72080]: 2026-06-24 11:19:17.238 WARNI [SnapshotInstaller] Snapshot download attempt 1/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:24:27.865 WARNI [SnapshotInstaller] Snapshot download attempt 2/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:24:27.865 WARNI [SnapshotInstaller] Snapshot download attempt 3/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:29:48.493 WARNI [SnapshotInstaller] Snapshot download attempt 4/4 failed for 'graph-data': Premature EOF
 java[72080]: 2026-06-24 11:29:49.181 SEVER [ArcadeStateMachine] Snapshot download triggered by watchdog failed
 java[72080]: java.io.IOException: Snapshot download failed after 4 attempts for 'graph-data'
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:522)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:185)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.install(SnapshotInstaller.java:137)
 java[72080]:         at com.arcadedb.server.ha.raft.ArcadeStateMachine.triggerSnapshotDownload(ArcadeStateMachine.java:1265)
 java[72080]:         at com.arcadedb.server.ha.raft.ArcadeStateMachine.lambda$applyBootstrapFingerprintEntry$6(ArcadeStateMachine.java:1076)
 java[72080]:         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
 java[72080]:         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
 java[72080]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 java[72080]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 java[72080]:         at java.base/java.lang.Thread.run(Thread.java:1583)
 java[72080]: Caused by: java.io.IOException: Premature EOF
 java[72080]:         at java.base/sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:567)
 java[72080]:         at java.base/sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:611)
 java[72080]:         at java.base/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:705)
 java[72080]:         at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
 java[72080]:         at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3707)
 java[72080]:         at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller$CountingInputStream.read(SnapshotInstaller.java:689)
 java[72080]:         at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
 java[72080]:         at java.base/java.io.PushbackInputStream.read(PushbackInputStream.java:197)
 java[72080]:         at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:264)
 java[72080]:         at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175)
 java[72080]:         at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:412)
 java[72080]:         at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.copyWithLimit(SnapshotInstaller.java:650)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadSnapshot(SnapshotInstaller.java:586)
 java[72080]:         at com.arcadedb.server.ha.raft.SnapshotInstaller.downloadWithRetry(SnapshotInstaller.java:512)
 java[72080]:         ... 9 more

Log from the leader

java[115047]: 2026-06-24 11:26:20.599 INFO  [SnapshotHttpHandler] Serving database snapshot for 'graph-data'...
java[115047]: 2026-06-24 11:29:48.489 WARNI [SnapshotHttpHandler] Snapshot write for 'graph-data' timed out after 300000ms, closing connection to release semaphore slot
java[115047]: 2026-06-24 11:29:48.491 SEVER [SnapshotHttpHandler] Error serving snapshot for 'graph-data'
java[115047]: java.io.IOException: UT000034: Stream is closed
java[115047]:         at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]:         at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]:         at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:232)
java[115047]:         at java.base/java.util.zip.ZipOutputStream.write(ZipOutputStream.java:361)
java[115047]:         at java.base/java.io.InputStream.transferTo(InputStream.java:797)
java[115047]:         at java.base/java.io.FileInputStream.transferTo(FileInputStream.java:390)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.addFileToZip(SnapshotHttpHandler.java:318)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:283)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$2(SnapshotHttpHandler.java:169)
java[115047]:         at com.arcadedb.utility.CodeUtils.executeIgnoringExceptions(CodeUtils.java:39)
java[115047]:         at com.arcadedb.engine.PageManager.suspendFlushAndExecute(PageManager.java:147)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$3(SnapshotHttpHandler.java:167)
java[115047]:         at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1730)
java[115047]:         at com.arcadedb.server.ha.raft.RaftReplicatedDatabase.executeInReadLock(RaftReplicatedDatabase.java:1023)
java[115047]:         at com.arcadedb.server.ServerDatabase.executeInReadLock(ServerDatabase.java:597)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.handleRequest(SnapshotHttpHandler.java:161)
java[115047]:         at io.undertow.server.Connectors.executeRootHandler(Connectors.java:368)
java[115047]:         at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:935)
java[115047]:         at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
java[115047]:         at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
java[115047]:         at java.base/java.lang.Thread.run(Thread.java:1583)
java[115047]:         Suppressed: java.io.IOException: UT000034: Stream is closed
java[115047]:                 at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]:                 at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:283)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:388)
java[115047]:                 at java.base/java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:267)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.close(ZipOutputStream.java:405)
java[115047]:                 at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:269)
java[115047]:                 ... 17 more
java[115047]: 2026-06-24 11:29:48.491 SEVER [CodeUtils] Error during suspend flush
java[115047]: java.lang.RuntimeException: java.io.IOException: UT000034: Stream is closed
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:300)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$2(SnapshotHttpHandler.java:169)
java[115047]:         at com.arcadedb.utility.CodeUtils.executeIgnoringExceptions(CodeUtils.java:39)
java[115047]:         at com.arcadedb.engine.PageManager.suspendFlushAndExecute(PageManager.java:147)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.lambda$handleRequest$3(SnapshotHttpHandler.java:167)
java[115047]:         at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1730)
java[115047]:         at com.arcadedb.server.ha.raft.RaftReplicatedDatabase.executeInReadLock(RaftReplicatedDatabase.java:1023)
java[115047]:         at com.arcadedb.server.ServerDatabase.executeInReadLock(ServerDatabase.java:597)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.handleRequest(SnapshotHttpHandler.java:161)
java[115047]:         at io.undertow.server.Connectors.executeRootHandler(Connectors.java:368)
java[115047]:         at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:935)
java[115047]:         at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
java[115047]:         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
java[115047]:         at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
java[115047]:         at java.base/java.lang.Thread.run(Thread.java:1583)
java[115047]: Caused by: java.io.IOException: UT000034: Stream is closed
java[115047]:         at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]:         at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]:         at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:232)
java[115047]:         at java.base/java.util.zip.ZipOutputStream.write(ZipOutputStream.java:361)
java[115047]:         at java.base/java.io.InputStream.transferTo(InputStream.java:797)
java[115047]:         at java.base/java.io.FileInputStream.transferTo(FileInputStream.java:390)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.addFileToZip(SnapshotHttpHandler.java:318)
java[115047]:         at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:283)
java[115047]:         ... 17 more
java[115047]:         Suppressed: java.io.IOException: UT000034: Stream is closed
java[115047]:                 at io.undertow.io.UndertowOutputStream.write(UndertowOutputStream.java:120)
java[115047]:                 at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:284)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:283)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:388)
java[115047]:                 at java.base/java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:267)
java[115047]:                 at java.base/java.util.zip.ZipOutputStream.close(ZipOutputStream.java:405)
java[115047]:                 at com.arcadedb.server.ha.raft.SnapshotHttpHandler.serveSnapshotZip(SnapshotHttpHandler.java:269)
java[115047]:                 ... 17 more

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions