Background

Already now, the OpenSearch project systematically conducts benchmark test runs and publishes the results: https://opensearch.org/benchmarks/ This is based on the pre-defined benchmark-workloads which are executed using the tool opensearch-benchmark.

Additionally, there is ongoing work to define testing processes for performance related quality assurance: opensearch-project/OpenSearch#7499

These efforts, however, focus on the OpenSearch core; the efficiency of the code implementing data retrieval and indexing is tested. In some cases, there is a comparison between OpenSearch with security enabled vs OpenSearch without security enabled. While this can give a first idea of the overhead produced by the security plugin, the results cannot be generalized. This is because the performance overhead of the security plugin depends on a number of different factors. See below for more information on this.

Ideally, there would be an additional security-focused performance test suite. This test suite could be used for assuring that the security plugin does not suffer any performance regressions. Additionally, it could be used to determine the actual benefit of performance enhancing code changes (for example #3870).

Goal of this RFC is to collect requirements for such a test suite, learn about ongoing efforts, and to find a way to integrate it into the software development process.

Performance influencing factors

The performance overhead of the security plugin depends on a number of different factors:

• Number of indices on a cluster. The security plugin access controls involve resolving requested index patterns and index patterns in the role definitions to actual indices. For clusters with many indices, this can slow things down (see for example Optimized Privilege Evaluation #3870 )

• Number of roles of a user. While the example configurations usually demonstrate users with one or two roles, there are environments which act very liberal in assigning roles to users. Users with hundreds of roles do happen in reality. In order to determine the authorization status of a user, the security plugin needs to loop through every role of a user for each request.

• Size of the user object. During inter-node communication, the security plugin needs to pass on the user object to the target node; for this, the user object needs to be serialized and deserialized. Larger user objects cause higher overhead during serialization and deserialization. Additionally, larger user objects put more stress onto the network.

• Use of patterns vs constant index names in the role configuration.

• Use of dynamic index patterns. The security plugin allows to define roles which use expressions to dynamically define the indices for which privileges are granted. Additionally, date math can be used to dynamically grant privileges. Both cases add a certain overhead to the privilege evaluation.

• Operations on aliases vs operations on indices. Privilege evaluation for operations on aliases require an additional resolving of the aliases. This can have a significant impact if an alias has many member indices.

• Use of DLS, FLS or field masking. If DLS, FLS or field masking is used, the security plugin hooks at a low level into the OpenSearch/Lucene operations. This is particularly hot code where already subtle details can have big performance impacts. Additionally, the current DLS/FLS implementation requires the serialization/deserialization of the whole DLS/FLS configuration. This can also have performance impacts for clusters with complex configurations.

• Type of the OpenSearch operation. While the operation itself (such as bulk index, search or aggregate) should have little effect on the performance, there can be operation specific optimizations which then lead to performance characteristics that vary by operation. For example, the privilege evaluation code contains special cases to skip it on the top level for bulk index operations in favour of checks on the shard level.

• TLS version, ciphers, implementation. TLS offers actually a quite broad field of different settings and implementations which might have an effect on the performance. As the security plugin adds TLS to the communication, it might be good to gain more insights into its performance effects.

Proposal for test dimensions

Given the complex dependencies, a three dimensional test matrix seems to be necessary. I would propose these dimensions:

1. Operations: bulk index, search on specific index, search on *, search on alias, search with aggregation
2. Number of indices: Cluster with 10 indices, cluster with 1000 indices
3. Security configuration: Basic, large user object, utilizing DLS/FLS, dynamic index patterns, user with many roles

Note: Some combinations in this matrix might not make sense and can be skipped; for example, DLS/FLS makes indices read-only, thus testing indexing does not make sense in that case

Questions:

• Are there further aspects that should be covered?
• Should a comparison of TLS versions, ciphers or implementations be also covered here?

Process definition

We need to define when and how the performance tests should be executed.

Factors which might influence the decision are:

• A test run will take a significant time. Thus, it might be not suitable to be included in a pull request CI run. Should there be an optional way to execute the test for a pull request?
• A test run will require significant CPU resources. Is the Github actions environment sufficient for (small) test runs? Is it necessary to pull up dedicated environments?
• Shall there be a nightly test run?
• How shall the results of the test runs be published?
• How shall the results of the test runs be monitored?

Related work

There are a number of other ongoing efforts/proposals covering OpenSearch benchmarking. In some cases, it might make sense to coordinate with these efforts:

• [RFC] OpenSearch Performance Testing Proposal OpenSearch#7499
• [RFC] Control plane for submitting benchmark runs. opensearch-build#4231
• Add a sample setup for performance benchmarking in security plugin #2896
• OpenSearch Performance Experiments Results OpenSearch#2461

Question: Are there further efforts?