[BUG][Performance] Permission Check needs many CPU Resources for User with all_access/admin #3104
Description
What is the bug?
Migrating from Opensearch 2.9 with disabled security to Opensearch 2.9 with enabled security we see a huge rise in cpu usage on our coordinating nodes.
How can one reproduce the bug?
Steps to reproduce the behavior:
- Switch from Opensearch w/o security to security
- Ingest heavy amounts of data with a user which has all_access
- Observe high cpu usage in coordinating nodes (2-10% without cpu usage -> 100%+ with security)
- High latency for all requests going to OS
What is the expected behavior?
Fast permission Checks if user has all_access
Possible Solution would be to add here a quick check if user is admin and then return true here:
Could be extended to some build in roles. (e.g. all actions are write actions -> user has role 'write' return permitted)
Audit Logging should not be affected by this as there are indices resolved again. (Auditing was disabled while enabling security)
What is your host/environment?
- OS: RHEL
- Opensearch Version 2.9
- Plugins vanilla Opensearch package (.tgz - https://opensearch.org/versions/opensearch-2-9-0.html )
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Output of hot_threads of one of the coordinating nodes:
47.3% (236.6ms out of 500ms) cpu usage by thread 'opensearch[hostname][transport_worker][T#12]'
6/10 snapshots sharing following 110 elements
app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:233)
app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:160)
org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:827)
org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:791)
org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1062)
org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5257/0x00000008013f3040.apply(Unknown Source)
java.base@11.0.16.1/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
java.base@11.0.16.1/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1621)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.base@11.0.16.1/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1063)
org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:516)
org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:524)
org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)