Native code has been identified as one major tool to enhance performance in critical parts of Elasticsearch.
Elasticsearch provides extensive tools and support for quality assurance of Java code; we need to have something similar to support compilation, testing, benchmarking, performance tuning, packaging and distribution of native code.

Some things are more complicated when dealing with native code: Java can be compiled and run anywhere, with the expectation that it will behave similarly (or close enough) across all supported platforms and processors. For native code, things like compilation and packaging are need to be created for all platforms we support; moreover, things like testing and benchmarking need to run across multiple processors, across multiple vendors of the same architecture and across multiple generations of the same architecture (e.g. AMD with AVX2 support, Intel with AVX-512 support, Graviton with Neon support, etc.).

We have already something in place: our native library can be compiled across all platforms (OS + architecture) on a single Mac, using docker, but it's a very manual process (you need to compile, test, benchmark across multiple platforms, juggling multiple cloud instances, then you can compile and upload the final binaries). We need to expand and improve this tooling.

• Expand the bulk vector search micro benchmarks to include dataset larger than typical cache sizes. #138358
• Improve tests / our test framework to simplify and increase test coverage of native code #139059
• Automate native code build/artifact publishing
• Automate JMH benchmarks runs across all supported processors
• Automate native code test runs across all supported processors