Skip to content

Refactored config REST API tests to avoid unsafe patterns causing memory leaks#5934

Merged
DarshitChanpura merged 1 commit intoopensearch-project:mainfrom
nibix:api-test-fix-refactoring
Feb 6, 2026
Merged

Refactored config REST API tests to avoid unsafe patterns causing memory leaks#5934
DarshitChanpura merged 1 commit intoopensearch-project:mainfrom
nibix:api-test-fix-refactoring

Conversation

@nibix
Copy link
Copy Markdown
Collaborator

@nibix nibix commented Jan 31, 2026

Description

In #5915, we discovered that the config REST API integration tests cause memory leaks as they use static references across test class boundaries in an unsafe manner.

This PR refactores these tests so that these no longer share static cluster or mutable static configuration references across test classes. IMHO, one should strive for abolishing the abstract super classes for these tests alltogether, but this is outside of my time availability.

  • Category: Test fix
  • Why these changes are required?
    • Fixes memory leaks in integration tests
  • What is the old behavior before changes and new behavior after changes?
    • No behavioral changes

Issues Resolved

Fixes #5915

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Nils Bandener <nils.bandener@eliatra.com>
@nibix
Copy link
Copy Markdown
Collaborator Author

nibix commented Feb 2, 2026

@cwperks @willyborankin

This should fix the remaining memory leak test issues. This is a big change, but it is a quite schematic code change.

Copy link
Copy Markdown
Member

@cwperks cwperks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great @nibix! Great analysis finding the shared static across test suites causing too much memory to be consumed.

@cwperks
Copy link
Copy Markdown
Member

cwperks commented Feb 2, 2026

@nibix Any idea why this shows up ~700 times in the output?

2026-01-31T16:09:15.6734273Z 2026-01-31 16:09:15 opensearch[cluster_manager_0][ConfigurationRepository#ReloadThread][T#1] ERROR AbstractRuleBasedPrivileges:644 - Unexpected exception while processing role: new_role_for_patch=RoleV7 [reserved=false, hidden=true, _static=false, description=null, cluster_permissions=[a, b, c], index_permissions=[Index [index_patterns=[ARoqhKffJL], dls=str1234567, fls=[aiZ1yaPda5, 7Dst9H8EK4], masked_fields=[zF4tUpaM6G, IvOISkCct1], allowed_actions=[J5qGGqhMTe, wNanvnOHhp, 8IH7tzLbfr, 9siK7L6Wxe, qsxsQvTD82]]], tenant_permissions=[]]
2026-01-31T16:09:15.6737247Z Ignoring role.
2026-01-31T16:09:15.6738005Z org.opensearch.security.privileges.PrivilegesConfigurationValidationException: Invalid DLS query: str1234567
2026-01-31T16:09:15.6739256Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery.parseQuery(DocumentPrivileges.java:132)
2026-01-31T16:09:15.6740531Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery$Constant.<init>(DocumentPrivileges.java:154)
2026-01-31T16:09:15.6741816Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery.create(DocumentPrivileges.java:141)
2026-01-31T16:09:15.6743093Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges.roleToRule(DocumentPrivileges.java:66)
2026-01-31T16:09:15.6744299Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges.lambda$new$0(DocumentPrivileges.java:57)
2026-01-31T16:09:15.6745692Z 	at org.opensearch.security.privileges.dlsfls.AbstractRuleBasedPrivileges$StaticRules.roleToRule(AbstractRuleBasedPrivileges.java:660)
2026-01-31T16:09:15.6747287Z 	at org.opensearch.security.privileges.dlsfls.AbstractRuleBasedPrivileges$StaticRules.<init>(AbstractRuleBasedPrivileges.java:619)
2026-01-31T16:09:15.6748706Z 	at org.opensearch.security.privileges.dlsfls.AbstractRuleBasedPrivileges.<init>(AbstractRuleBasedPrivileges.java:108)
2026-01-31T16:09:15.6749944Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges.<init>(DocumentPrivileges.java:57)
2026-01-31T16:09:15.6751104Z 	at org.opensearch.security.privileges.dlsfls.DlsFlsProcessedConfig.<init>(DlsFlsProcessedConfig.java:45)
2026-01-31T16:09:15.6752291Z 	at org.opensearch.security.configuration.DlsFlsValveImpl.updateConfiguration(DlsFlsValveImpl.java:772)
2026-01-31T16:09:15.6753535Z 	at org.opensearch.security.OpenSearchSecurityPlugin.lambda$createComponents$13(OpenSearchSecurityPlugin.java:1243)
2026-01-31T16:09:15.6754932Z 	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:567)
2026-01-31T16:09:15.6756463Z 	at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:556)
2026-01-31T16:09:15.6757967Z 	at org.opensearch.security.configuration.ConfigurationRepository.doReload(ConfigurationRepository.java:551)
2026-01-31T16:09:15.6759276Z 	at org.opensearch.security.configuration.ConfigurationRepository$ReloadThread.run(ConfigurationRepository.java:854)
2026-01-31T16:09:15.6760191Z 	at java.base/java.lang.Thread.run(Thread.java:1583)

ref: https://productionresultssa12.blob.core.windows.net/actions-results/559040d0-502f-440a-a784-dfe1a691bb91/workflow-job-run-1ce71013-3699-5650-a725-6bcf26eba9fb/logs/job/job-logs.txt?rsct=text%2Fplain&se=2026-02-02T15%3A01%3A13Z&sig=XOvk9nncV85HZW0awXOEsLC8xVDrb53EMjBnK4q1rDY%3D&ske=2026-02-02T17%3A45%3A54Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2026-02-02T13%3A45%3A54Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2026-02-02T14%3A51%3A08Z&sv=2025-11-05

Edit:

More of the stack trace:

2026-01-31T16:09:15.6686247Z Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'str1234567': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
2026-01-31T16:09:15.6687902Z  at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 11]
2026-01-31T16:09:15.6689046Z 	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2599)
2026-01-31T16:09:15.6690121Z 	at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2625)
2026-01-31T16:09:15.6691268Z 	at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2633)
2026-01-31T16:09:15.6692491Z 	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:831)
2026-01-31T16:09:15.6693992Z 	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:3017)
2026-01-31T16:09:15.6695536Z 	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2051)
2026-01-31T16:09:15.6727135Z 	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:780)
2026-01-31T16:09:15.6728320Z 	at org.opensearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:66)
2026-01-31T16:09:15.6729476Z 	at org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:336)
2026-01-31T16:09:15.6730727Z 	at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery.parseQuery(DocumentPrivileges.java:130)
2026-01-31T16:09:15.6731528Z 	... 16 more

@cwperks
Copy link
Copy Markdown
Member

cwperks commented Feb 2, 2026

Looks like the DLS clause in new_role_for_patch is malformed. We should fix that, but could be done separately from this PR.

@nibix
Copy link
Copy Markdown
Collaborator Author

nibix commented Feb 2, 2026

Looks like the DLS clause in new_role_for_patch is malformed. We should fix that, but could be done separately from this PR.

Before we removed the randomized testing framework, these were just random strings. I was under the impression that validation was tested here, but I am not sure.

@cwperks
Copy link
Copy Markdown
Member

cwperks commented Feb 3, 2026

@willyborankin @DarshitChanpura can we get another review on this?

Copy link
Copy Markdown
Member

@DarshitChanpura DarshitChanpura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @nibix .

@DarshitChanpura DarshitChanpura merged commit f56424b into opensearch-project:main Feb 6, 2026
254 of 260 checks passed
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 6, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (e842cd2) to head (5e12d9b).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main   #5934       +/-   ##
==========================================
- Coverage   73.83%       0   -73.84%     
==========================================
  Files         439       0      -439     
  Lines       26884       0    -26884     
  Branches     3980       0     -3980     
==========================================
- Hits        19851       0    -19851     
+ Misses       5148       0     -5148     
+ Partials     1885       0     -1885     

see 439 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Tests] Integration test issues

3 participants