DedicatedClusterSnapshotRestoreIT.testMasterAndDataShutdownDuringSnapshot failed in https://gradle-enterprise.elastic.co/s/ollsfmdiyy7ts/console-log/raw with the following:
REPRODUCE WITH: ./gradlew ':server:integTest' --tests "org.elasticsearch.snapshots.DedicatedClusterSnapshotRestoreIT.testMasterAndDataShutdownDuringSnapshot" -Dtests.seed=4090EEDAC757A077 -Dtests.security.manager=true -Dtests.locale=pl-PL -Dtests.timezone=Pacific/Tarawa -Dcompiler.java=12
org.elasticsearch.snapshots.DedicatedClusterSnapshotRestoreIT > testMasterAndDataShutdownDuringSnapshot FAILED
org.elasticsearch.snapshots.SnapshotMissingException: [test-repo:test-snap] is missing
It seems the problem is here.
1> [2019-12-10T02:00:32,554][WARN ][o.e.s.SnapshotsService ] [node_tm0] failed to update snapshot state after node removal
1> [2019-12-10T02:00:32,558][WARN ][o.e.s.SnapshotsService ] [node_tm0] [test-repo:test-snap/m5-vYhZeSIm67jm7Wz-3ow] failed to finalize snapshot
1> org.elasticsearch.snapshots.SnapshotException: [test-repo:test-snap/m5-vYhZeSIm67jm7Wz-3ow] failed to update snapshot in repository
1> at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$31(BlobStoreRepository.java:812) [main/:?]
1> at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [main/:?]
1> at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.onFailure(BlobStoreRepository.java:1118) [main/:?]
1> at org.elasticsearch.cluster.ClusterStateTaskListener.onNoLongerMaster(ClusterStateTaskListener.java:37) [main/:?]
1> at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.onNoLongerMaster(MasterService.java:519) [main/:?]
1> at org.elasticsearch.cluster.service.MasterService$TaskInputs.lambda$onNoLongerMaster$0(MasterService.java:762) [main/:?]
1> at java.util.ArrayList.forEach(ArrayList.java:1540) [?:?]
1> at org.elasticsearch.cluster.service.MasterService$TaskInputs.onNoLongerMaster(MasterService.java:762) [main/:?]
1> at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:209) [main/:?]
1> at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [main/:?]
1> at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [main/:?]
1> at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:629) [main/:?]
1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [main/:?]
1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
1> at java.lang.Thread.run(Thread.java:834) [?:?]
1> Caused by: org.elasticsearch.repositories.RepositoryException: [test-repo] Failed to execute cluster state update [set pending repository generation [test-repo][-1]]
1> at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.onFailure(BlobStoreRepository.java:1119) ~[main/:?]
1> ... 15 more
1> Caused by: org.elasticsearch.cluster.NotMasterException: no longer master. source: [set pending repository generation [test-repo][-1]]
It seems the test does not fully take the fact that we need to form a working cluster for the snapshot to finalise due to #49729 nowadays. I'm looking into that.
DedicatedClusterSnapshotRestoreIT.testMasterAndDataShutdownDuringSnapshotfailed in https://gradle-enterprise.elastic.co/s/ollsfmdiyy7ts/console-log/raw with the following:It seems the problem is here.
It seems the test does not fully take the fact that we need to form a working cluster for the snapshot to finalise due to #49729 nowadays. I'm looking into that.