[Snapshot Interop][Bug Fix] Ensure lock file gets deleted if snapshot … by harishbhakuni · Pull Request #12016 · opensearch-project/OpenSearch

harishbhakuni · 2024-01-25T01:31:48Z

...shard md upload fails during snapshot creation.

Description

While working on another change, realized that in the following code, everything below snapshotRemoteStoreIndexShard call was not getting executed:

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java

Lines 427 to 462 in 829215c

    
               repository.snapshotRemoteStoreIndexShard( 
        
                   indexShard.store(), 
        
                   snapshot.getSnapshotId(), 
        
                   indexId, 
        
                   snapshotIndexCommit, 
        
                   getShardStateId(indexShard, snapshotIndexCommit), 
        
                   snapshotStatus, 
        
                   primaryTerm, 
        
                   startTime, 
        
                   ActionListener.runBefore(listener, wrappedSnapshot::close) 
        
               ); 
        
           } catch (IndexShardSnapshotFailedException e) { 
        
               logger.error( 
        
                   "Shallow Copy Snapshot Failed for Shard [" 
        
                       + indexId.getName() 
        
                       + "][" 
        
                       + shardId.getId() 
        
                       + "] for snapshot " 
        
                       + snapshot.getSnapshotId() 
        
                       + ", releasing acquired lock from remote store" 
        
               ); 
        
               indexShard.releaseLockOnCommitData(snapshot.getSnapshotId().getUUID(), primaryTerm, commitGeneration); 
        
               throw e; 
        
           } 
        
           long endTime = threadPool.relativeTimeInMillis(); 
        
           logger.debug( 
        
               "Time taken (in milliseconds) to complete shallow copy snapshot, " 
        
                   + "for index " 
        
                   + indexId.getName() 
        
                   + ", shard " 
        
                   + shardId.getId() 
        
                   + " and snapshot " 
        
                   + snapshot.getSnapshotId() 
        
                   + " is " 
        
                   + (endTime - startTime) 
        
           );

that was causing an issue where if shard md upload to snapshot repository fails, it will not release the lock file from S3.
as part of this PR, making sure those lines of code get executed even if shard md upload fails and adding IT to cover that scenario.
Also, Adding a log line to print the time taken in flush operation during snapshot creation and another minor change to filter lock files based on lock suffix while fetching lock files from lock directory.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-01-25T01:50:22Z

Compatibility status:

Checks if related components are compatible with change 197ccfd

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git]

github-actions · 2024-01-25T02:19:31Z

❌ Gradle check result for b8ec898: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harishbhakuni · 2024-01-25T04:46:18Z

Test org.opensearch.search.sort.FieldSortIT.testSimpleSorts {p0={"search.concurrent_segment_search.enabled":"true"}} is failing which is flaky and being tracked as part of #11875 .

opensearch-trigger-bot · 2024-05-09T15:21:47Z

This PR is stalled because it has been open for 30 days with no activity.

github-actions · 2024-06-04T22:16:49Z

❌ Gradle check result for 4cdd33e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harishbhakuni · 2024-06-04T22:20:09Z

org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction is flaky: #12197

github-actions · 2024-06-04T23:14:33Z

❌ Gradle check result for 203e404: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harishbhakuni · 2024-06-05T01:03:05Z

#13600
org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}},
org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"true"}},
org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"true"}},
org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"false"}}

#13939
org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testRemoteIndexPathFileExistsAfterMigration:

github-actions · 2024-06-05T20:09:00Z

❌ Gradle check result for 184132e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harishbhakuni · 2024-06-05T21:19:32Z

org.opensearch.indices.IndicesRequestCacheIT.classMethod: test suite timed out.

All the below mentioned tests are flaky:
org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndexFailure: #13476
org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock: #10006
org.opensearch.indices.IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"false"}}
org.opensearch.indices.IndicesRequestCacheIT.classMethod: #13437

github-actions · 2024-06-06T04:58:46Z

❌ Gradle check result for 1ebbc06: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

harishbhakuni · 2024-06-06T14:39:01Z

org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndexFailure: #13476
org.opensearch.indices.IndicesRequestCacheIT.testCacheCleanupWithDefaultSettings {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"true"}}
org.opensearch.indices.IndicesRequestCacheIT.classMethod: #13711

…hot shard md upload fails during snapshot creation. Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

github-actions · 2024-06-06T19:04:44Z

❌ Gradle check result for 57799ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

gbbafna · 2024-06-11T09:55:36Z

"that was causing an issue where if shard md upload to snapshot repository fails, it will not release the lock file from S3." - How is that happening ?

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java

Lines 438 to 462 in 829215c

    
           } catch (IndexShardSnapshotFailedException e) { 
        
               logger.error( 
        
                   "Shallow Copy Snapshot Failed for Shard [" 
        
                       + indexId.getName() 
        
                       + "][" 
        
                       + shardId.getId() 
        
                       + "] for snapshot " 
        
                       + snapshot.getSnapshotId() 
        
                       + ", releasing acquired lock from remote store" 
        
               ); 
        
               indexShard.releaseLockOnCommitData(snapshot.getSnapshotId().getUUID(), primaryTerm, commitGeneration); 
        
               throw e; 
        
           } 
        
           long endTime = threadPool.relativeTimeInMillis(); 
        
           logger.debug( 
        
               "Time taken (in milliseconds) to complete shallow copy snapshot, " 
        
                   + "for index " 
        
                   + indexId.getName() 
        
                   + ", shard " 
        
                   + shardId.getId() 
        
                   + " and snapshot " 
        
                   + snapshot.getSnapshotId() 
        
                   + " is " 
        
                   + (endTime - startTime) 
        
           );

The above block is doing exactly that

harishbhakuni · 2024-06-11T20:28:49Z

The above block is doing exactly that

not really, the problem was that caller method snapshotRemoteStoreIndexShard was catching the exception and passing it with listener.onFailure() method. so the catch block where we were releasing lock will never get executed. just moved that code to onFailure() method. so that it will get executed.

gbbafna · 2024-06-12T11:28:46Z

server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java

-                            + snapshot.getSnapshotId()
-                            + " is "
-                            + (endTime - startTime)
+                    GatedCloseable<IndexCommit> finalWrappedSnapshot = wrappedSnapshot;


we should throw IndexShardSnapshotFailedException along with listener.onFailure(e);

OpenSearch/server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

Line 2691 in 829215c

listener.onFailure(e);

gbbafna · 2024-06-12T11:55:03Z

server/src/main/java/org/opensearch/index/store/lockmanager/RemoteStoreMetadataLockManager.java

+            return new String[0];
+        }
+        // filtering lock files from lock directory contents.
+        // this is a good to have check, there is no known prod scenarios where this can happen


nit : lets not use prod word . Also a user can create extra files in the lock folder as well

andrross · 2025-08-27T22:29:38Z

@harishbhakuni Is this still an issue? Can we update or close this PR?

harishbhakuni self-assigned this Jan 25, 2024

harishbhakuni added the skip-changelog label Jan 25, 2024

harishbhakuni changed the title ~~[Snapshot Interop][Bug Fix] Make sure lock file gets deleted if snaps…~~ [Snapshot Interop][Bug Fix] Ensure lock file gets deleted if snapshot … Jan 25, 2024

harishbhakuni marked this pull request as ready for review January 25, 2024 04:46

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Apr 9, 2024

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels May 9, 2024

harishbhakuni force-pushed the interopBugFix branch from 197ccfd to 4cdd33e Compare June 4, 2024 21:20

harishbhakuni force-pushed the interopBugFix branch from 4cdd33e to 203e404 Compare June 4, 2024 22:21

harishbhakuni force-pushed the interopBugFix branch from 203e404 to 184132e Compare June 5, 2024 19:18

harishbhakuni force-pushed the interopBugFix branch from 184132e to 1ebbc06 Compare June 6, 2024 04:10

[Snapshot Interop][Bug Fix] Make sure lock file gets deleted if snaps…

57799ac

…hot shard md upload fails during snapshot creation. Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

harishbhakuni force-pushed the interopBugFix branch from 1ebbc06 to 57799ac Compare June 6, 2024 18:11

gbbafna reviewed Jun 12, 2024

View reviewed changes

prudhvigodithi mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

	repository.snapshotRemoteStoreIndexShard(
	indexShard.store(),
	snapshot.getSnapshotId(),
	indexId,
	snapshotIndexCommit,
	getShardStateId(indexShard, snapshotIndexCommit),
	snapshotStatus,
	primaryTerm,
	startTime,
	ActionListener.runBefore(listener, wrappedSnapshot::close)
	);
	} catch (IndexShardSnapshotFailedException e) {
	logger.error(
	"Shallow Copy Snapshot Failed for Shard ["
	+ indexId.getName()
	+ "]["
	+ shardId.getId()
	+ "] for snapshot "
	+ snapshot.getSnapshotId()
	+ ", releasing acquired lock from remote store"
	);
	indexShard.releaseLockOnCommitData(snapshot.getSnapshotId().getUUID(), primaryTerm, commitGeneration);
	throw e;
	}
	long endTime = threadPool.relativeTimeInMillis();
	logger.debug(
	"Time taken (in milliseconds) to complete shallow copy snapshot, "
	+ "for index "
	+ indexId.getName()
	+ ", shard "
	+ shardId.getId()
	+ " and snapshot "
	+ snapshot.getSnapshotId()
	+ " is "
	+ (endTime - startTime)
	);

Conversation

harishbhakuni commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Check List

Uh oh!

github-actions bot commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

github-actions bot commented Jan 25, 2024

Uh oh!

harishbhakuni commented Jan 25, 2024

Uh oh!

opensearch-trigger-bot bot commented May 9, 2024

Uh oh!

github-actions bot commented Jun 4, 2024

Uh oh!

harishbhakuni commented Jun 4, 2024

Uh oh!

github-actions bot commented Jun 4, 2024

Uh oh!

harishbhakuni commented Jun 5, 2024

Uh oh!

github-actions bot commented Jun 5, 2024

Uh oh!

harishbhakuni commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 6, 2024

Uh oh!

harishbhakuni commented Jun 6, 2024

Uh oh!

github-actions bot commented Jun 6, 2024

Uh oh!

gbbafna commented Jun 11, 2024

Uh oh!

harishbhakuni commented Jun 11, 2024

Uh oh!

gbbafna Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

gbbafna Jun 12, 2024

Choose a reason for hiding this comment

Uh oh!

andrross commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

harishbhakuni commented Jan 25, 2024 •

edited

Loading

github-actions bot commented Jan 25, 2024 •

edited

Loading

harishbhakuni commented Jun 5, 2024 •

edited

Loading