Skip to content

Remove unnecessary iteration in request cache clear#19263

Merged
jainankitk merged 23 commits intoopensearch-project:mainfrom
peteralfonsi:irc-cache-clear-fix
Oct 1, 2025
Merged

Remove unnecessary iteration in request cache clear#19263
jainankitk merged 23 commits intoopensearch-project:mainfrom
peteralfonsi:irc-cache-clear-fix

Conversation

@peteralfonsi
Copy link
Copy Markdown
Contributor

Description

Adds a node-level hook to TransportBroadcastByNodeAction which runs after all shard-level operations on that node have finished.

Also reworks TransportClearIndicesCacheAction to use this new hook, to prevent needlessly iterating through all cache keys once per shard, which caused high cache clear latency especially if the disk cache is used. I bundled the two together into one PR since both changes are pretty small.

I got cache clear numbers from cleaning up 2M request cache keys in a disk tier, spread across 10 indices (1 shard each), on a single node hosted on a c5.2xl instance.

Run Clear time (sec)
Baseline 131.56
Contender 60.62

Related Issues

Resolves:
#19183
#19118

Check List

  • Functionality includes testing.
  • [N/A] API changes companion pull request created, if applicable.
  • [N/A] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Peter Alfonsi added 3 commits September 8, 2025 12:48
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@peteralfonsi peteralfonsi requested a review from a team as a code owner September 9, 2025 19:17
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 9, 2025

❌ Gradle check result for f650cb4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Peter Alfonsi and others added 2 commits September 9, 2025 14:15
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <peter.alfonsi@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 9, 2025

✅ Gradle check result for 9004301: SUCCESS

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 9, 2025

Codecov Report

❌ Patch coverage is 36.36364% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.89%. Comparing base (259add3) to head (350f34e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
.../cache/clear/TransportClearIndicesCacheAction.java 16.66% 5 Missing ⚠️
...in/java/org/opensearch/indices/IndicesService.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #19263      +/-   ##
============================================
+ Coverage     72.88%   72.89%   +0.01%     
- Complexity    70219    70246      +27     
============================================
  Files          5702     5702              
  Lines        322202   322216      +14     
  Branches      46625    46627       +2     
============================================
+ Hits         234832   234892      +60     
+ Misses        68418    68354      -64     
- Partials      18952    18970      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@peteralfonsi
Copy link
Copy Markdown
Contributor Author

Codecov is saying the methods for cache clear are not covered, this is not true as they are covered in the ITs (confirmed with IntelliJ debugger)

Signed-off-by: Peter Alfonsi <peter.alfonsi@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

❕ Gradle check result for 7d3cd10: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Peter Alfonsi <peter.alfonsi@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 37486b8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@peteralfonsi
Copy link
Copy Markdown
Contributor Author

Flaky tests: #19354, #17678

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for fa15ef0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@peteralfonsi
Copy link
Copy Markdown
Contributor Author

#17678

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 5e334a9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@peteralfonsi
Copy link
Copy Markdown
Contributor Author

#17678 again

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for d3081ee: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@peteralfonsi
Copy link
Copy Markdown
Contributor Author

Flaky tests: #14291, #16683, #15813

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 10aa089: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 668b8ea: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for fb5fe89: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 598cdf1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <peter.alfonsi@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for b43d86b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Ankit Jain <jainankitk@apache.org>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 1, 2025

❌ Gradle check result for 288fe1c: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 1, 2025

❕ Gradle check result for 350f34e: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@jainankitk jainankitk merged commit efe4877 into opensearch-project:main Oct 1, 2025
32 of 33 checks passed
peteralfonsi added a commit to peteralfonsi/OpenSearch that referenced this pull request Oct 15, 2025
…ct#19263)

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <peter.alfonsi@gmail.com>
Signed-off-by: Ankit Jain <jainankitk@apache.org>
Co-authored-by: Peter Alfonsi <petealft@amazon.com>
Co-authored-by: Ankit Jain <jainankitk@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants