I have a Storm (2.8.0) cluster with 3 Nimbus and 3 Supervisor servers and replication count 3.
When submitting a new topolgy I sometimes get the ERROR Could not download the blob with key: ....
The error occurs when the LocalFsBlobStoreSynchronizer determines the missing blob keys during synchronization. But the function BlobStoreUtils.downloadMissingBlob throws a KeyAlreadyExistsException, which is logged with INFO. In my opinion, isSucess should be set to true at this point and the loop exited. The actual error is logged after all other Nimbus servers have been iterated over and each iteration has ended unsuccessfully with a KeyAlreadyExistsException.
The KeyAlreadyExistsException is thrown because the corresponding blob file already exists in the file system.
I am not sure if either LocalFsBlobStoreSynchronizer.getBlobStoreKeySet() returns an inconsistent key set or the blob file has already been created in parallel.
I have attached the relevant Nimbus log.
I am grateful for any help.
blobstore_error.log
I have a Storm (2.8.0) cluster with 3 Nimbus and 3 Supervisor servers and replication count 3.
When submitting a new topolgy I sometimes get the ERROR
Could not download the blob with key: ....The error occurs when the
LocalFsBlobStoreSynchronizerdetermines the missing blob keys during synchronization. But the functionBlobStoreUtils.downloadMissingBlobthrows aKeyAlreadyExistsException, which is logged with INFO. In my opinion,isSucessshould be set totrueat this point and the loop exited. The actual error is logged after all other Nimbus servers have been iterated over and each iteration has ended unsuccessfully with aKeyAlreadyExistsException.The
KeyAlreadyExistsExceptionis thrown because the corresponding blob file already exists in the file system.I am not sure if either
LocalFsBlobStoreSynchronizer.getBlobStoreKeySet()returns an inconsistent key set or the blob file has already been created in parallel.I have attached the relevant Nimbus log.
I am grateful for any help.
blobstore_error.log