Skip to content

Fix security index creation race condition#128825

Merged
jfreden merged 3 commits intoelastic:mainfrom
jfreden:fix_jwt_test_issue
Jun 4, 2025
Merged

Fix security index creation race condition#128825
jfreden merged 3 commits intoelastic:mainfrom
jfreden:fix_jwt_test_issue

Conversation

@jfreden
Copy link
Copy Markdown
Contributor

@jfreden jfreden commented Jun 3, 2025

Resolves: #127694, #122863

The error occurs when trying to authenticate the activate profile request:

[2025-05-05T01:02:23,272][WARN ][o.e.x.s.a.RealmsAuthenticator][testActivateProfileForJWT] An error occurred while attempting to authenticate ['aud:es-01' 'groups:admin' 'iss:my-issuer-01' 'sub:me'] against realm [jwt0]
1> org.elasticsearch.action.UnavailableShardsException: at least one search shard for the index [.security-7] is unavailable

Before the test, the creation of the .security index is skipped:

[2025-05-05T01:02:23,174][INFO ][o.e.x.s.a.j.JwtRealmSingleNodeTests][testActivateProfileForJWT] Security index already exists, ignoring.

And before that we see:

[2025-05-05T01:02:23,127][INFO ][o.e.c.m.MetadataCreateIndexService][node_s_0][masterService#updateTask][T#1] creating index [.security-7] in project [default], cause [api], templates [], shards [1]/[1]
[2025-05-05T01:02:23,134][INFO ][o.e.c.r.a.AllocationService][node_s_0][masterService#updateTask][T#1] in project [default] updating number_of_replicas to [0] for indices [.security-7]

Even earlier we see:

[2025-05-05T01:02:22,854][INFO ][o.e.x.s.s.SecurityIndexManager][node_s_0][generic][T#6] security index does not exist, creating [.security-7] with alias [.security]

This is triggered by prepareIndexIfNeededThenExecute.

What I think is happening is that the index is created but not available and therefore it fails. If we check if exists and also wait for it to become available this might not happen.

@jfreden jfreden requested a review from slobodanadamovic June 3, 2025 13:07
@jfreden jfreden added >test Issues or PRs that are addressing/adding tests :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Jun 3, 2025
@elasticsearchmachine elasticsearchmachine added Team:Security Meta label for security team v9.1.0 labels Jun 3, 2025
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-security (Team:Security)

Copy link
Copy Markdown
Contributor

@slobodanadamovic slobodanadamovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Thanks for fixing this 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) Team:Security Meta label for security team >test Issues or PRs that are addressing/adding tests v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] JwtRealmSingleNodeTests testActivateProfileForJWT failing

3 participants