Fix entitlements in internalClusterTest by mosche · Pull Request #131539 · elastic/elasticsearch

mosche · 2025-07-18T15:12:46Z

Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive).
The shared data dir is granted as additional data base directory.
Due to the lack of entitlement delegation and wipePendingDataDirectories using server's FileSystemUtils, node base directories won't be removed until after the test.
Disable entitlement checks for some command tests.
Disable entitlement checks for some tests requiring entitlement delegation.

I'll follow up with better managing the lifecycle of test entitlement state, as discussed on Slack.

* Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). * The shared data dir is granted as additional data base directory. * Due to the lack of entitlement delegation and wipePendingDataDirectories using server's FileSystemUtils, node base directories won't be removed until after the test. * Disable entitlement checks for some command tests. * Disable entitlement checks for some tests requiring entitlement delegation.

elasticsearchmachine · 2025-07-18T15:13:11Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/policy/PolicyManager.java

mosche · 2025-07-18T15:16:50Z

...ramework/src/main/java/org/elasticsearch/entitlement/bootstrap/TestEntitlementBootstrap.java

            : dataDirs.stream().map(TestEntitlementBootstrap::absolutePath).toList();
    }

+    private static Path sharedDataDir(Settings settings) {


This is required for some tests, though it looks like we never grant PATH_SHARED_DATA_SETTING in production.
Is this a test-only thing? Or is that a bug?

prdoyle · 2025-07-18T15:25:08Z

...nternalClusterTest/java/org/elasticsearch/index/shard/RemoveCorruptedShardDataCommandIT.java

 import static org.hamcrest.Matchers.startsWith;

 @ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
+@ESTestCase.WithoutEntitlements // commands don't run with entitlements enforced


It's a bit unfortunate how often we need to do this. Makes me wonder if there's a more general rule we could apply so that WithoutEntitlements only needs to be used in exceptional cases. 🤔

prdoyle

We're on the same page about what's needed. I'll proactively approve to avoid delays.

… entitlement delegation

…xInternalClusterTest

Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). This change makes sure entitlements are correctly enabled during tests. Due to the lack of entitlement delegation (and usage of server's FileSystemUtils and similar in test code), there's a few remaining issues: - various tests have to run without entitlements - node base dirs cannot be removed immediately when shutting down the node due to pending cleanups (wipePendingDataDirectories) Due to Netty dependency issues (ES-12435), azure and inference tests have to run without entitlements.

elasticsearchmachine · 2025-07-29T12:03:11Z

💔 Backport failed

Status	Branch	Result
✅	9.1
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 131539

Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). This change makes sure entitlements are correctly enabled during tests. Due to the lack of entitlement delegation (and usage of server's FileSystemUtils and similar in test code), there's a few remaining issues: - various tests have to run without entitlements - node base dirs cannot be removed immediately when shutting down the node due to pending cleanups (wipePendingDataDirectories) Due to Netty dependency issues (ES-12435), azure and inference tests have to run without entitlements. (cherry picked from commit 5d72a3f) # Conflicts: # modules/repository-azure/src/internalClusterTest/java/org/elasticsearch/repositories/azure/AzureBlobStoreRepositoryTests.java

mosche · 2025-07-29T12:08:58Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). This change makes sure entitlements are correctly enabled during tests. Due to the lack of entitlement delegation (and usage of server's FileSystemUtils and similar in test code), there's a few remaining issues: - various tests have to run without entitlements - node base dirs cannot be removed immediately when shutting down the node due to pending cleanups (wipePendingDataDirectories) Due to Netty dependency issues (ES-12435), azure and inference tests have to run without entitlements.

Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). This change makes sure entitlements are correctly enabled during tests. Due to the lack of entitlement delegation (and usage of server's FileSystemUtils and similar in test code), there's a few remaining issues: - various tests have to run without entitlements - node base dirs cannot be removed immediately when shutting down the node due to pending cleanups (wipePendingDataDirectories) Due to Netty dependency issues (ES-12435), azure and inference tests have to run without entitlements. (cherry picked from commit 5d72a3f)

…cularityError. We've seen this previously (and predictably) in elastic#131539 due to logging Exceptions in isTriviallyAllowed. elastic#133269 shows exactly the same symptoms. Fixes elastic#133269 Fixes elastic#133267 Fixes elastic#133268

…cularityError. (#134431) We've seen this previously (and predictably) in #131539 due to logging Exceptions in isTriviallyAllowed. #133269 shows exactly the same symptoms. Fixes #133269 Fixes #133267 Fixes #133268

…cularityError. (elastic#134431) We've seen this previously (and predictably) in elastic#131539 due to logging Exceptions in isTriviallyAllowed. elastic#133269 shows exactly the same symptoms. Fixes elastic#133269 Fixes elastic#133267 Fixes elastic#133268

…cularityError. (#134431) (#134445) We've seen this previously (and predictably) in #131539 due to logging Exceptions in isTriviallyAllowed. #133269 shows exactly the same symptoms. Fixes #133269 Fixes #133267 Fixes #133268

…cularityError. (#134431) (#134444) We've seen this previously (and predictably) in #131539 due to logging Exceptions in isTriviallyAllowed. #133269 shows exactly the same symptoms. Fixes #133269 Fixes #133267 Fixes #133268

mosche requested a review from a team July 18, 2025 15:12

mosche added >refactoring test-windows Trigger CI checks on Windows :Core/Infra/Entitlements Entitlements infrastructure test-fips Trigger CI checks for FIPS labels Jul 18, 2025

elasticsearchmachine added v9.2.0 Team:Core/Infra Meta label for core/infra team labels Jul 18, 2025

mosche commented Jul 18, 2025

View reviewed changes

libs/entitlement/src/main/java/org/elasticsearch/entitlement/runtime/policy/PolicyManager.java Show resolved Hide resolved

mosche commented Jul 18, 2025

View reviewed changes

prdoyle reviewed Jul 18, 2025

View reviewed changes

prdoyle approved these changes Jul 18, 2025

View reviewed changes

mosche added 7 commits July 21, 2025 12:36

continue logging stacktrace in isTriviallyAllowed

3bf8256

Run Azure tests without entitlements until ES-12435

707e792

Run inference tests without entitlements until ES-12435

491f8b6

Merge branch 'main' into entitlements/fixInternalClusterTest

a43e9b1

Don't log exception in isTriviallyAllowed

02757bf

Merge branch 'main' into entitlements/fixInternalClusterTest

1373ee1

Grant file entitlements for shared data dir to ES base due to lack of…

ca4fd4b

… entitlement delegation

mosche mentioned this pull request Jul 23, 2025

Grant file entitlements for shared data dir to ES base due to lack of entitlement delegation #131748

Merged

mosche added 9 commits July 23, 2025 12:07

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

cadd799

…xInternalClusterTest

grant shared_data dir to lucene

eac1835

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

ea06394

…xInternalClusterTest

fix

b85aff5

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

36fa717

…xInternalClusterTest

fix

7b7a7f3

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

cc5fdeb

…xInternalClusterTest

logging

6351dde

grant shared_data dir to plugins that have a grant on data dir indices/

43f856e

mosche added 7 commits July 24, 2025 09:13

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

8af6bfa

…xInternalClusterTest

Merge branch 'main' into entitlements/shared_data_path_fix

d112237

Merge branch 'main' into entitlements/fixInternalClusterTest

e86c705

fix parser

35dfa0d

Merge branch 'main' into entitlements/shared_data_path_fix

87cc124

Merge branch 'entitlements/shared_data_path_fix' into entitlements/fi…

ffd993f

…xInternalClusterTest

Merge branch 'main' into entitlements/fixInternalClusterTest

475ecd9

mosche added the auto-backport Automatically create backport pull requests when merged label Jul 25, 2025

Merge branch 'main' into entitlements/fixInternalClusterTest

4479411

mosche added v9.1.1 v8.19.1 labels Jul 29, 2025

mosche merged commit 5d72a3f into elastic:main Jul 29, 2025
45 checks passed

mosche mentioned this pull request Jul 29, 2025

[9.1] Fix entitlements in internalClusterTest (#131539) #132067

Merged

elasticsearchmachine added the backport pending label Jul 29, 2025

mosche mentioned this pull request Jul 29, 2025

[8.19] Fix entitlements in internalClusterTest (#131539) #132070

Merged

mosche mentioned this pull request Sep 10, 2025

Remove logging of trusted system classes to not risk causing ClassCircularityError #134431

Merged

mosche removed the backport pending label Nov 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix entitlements in internalClusterTest#131539

Fix entitlements in internalClusterTest#131539
mosche merged 25 commits intoelastic:mainfrom
mosche:entitlements/fixInternalClusterTest

mosche commented Jul 18, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

Uh oh!

mosche Jul 18, 2025

Uh oh!

prdoyle Jul 18, 2025

Uh oh!

prdoyle left a comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 29, 2025

Uh oh!

mosche commented Jul 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mosche commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

Uh oh!

mosche Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

prdoyle Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

prdoyle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 29, 2025

💔 Backport failed

Uh oh!

mosche commented Jul 29, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mosche commented Jul 18, 2025 •

edited

Loading