Parameterize the vector result benchmarks on implementation and function#139423
Parameterize the vector result benchmarks on implementation and function#139423thecoop merged 8 commits intoelastic:mainfrom
Conversation
|
Pinging @elastic/es-search-relevance (Team:Search Relevance) |
ldematte
left a comment
There was a problem hiding this comment.
I like it! Some small comments but look good to me
| float vec2CorrectionConstant; | ||
| float scoreCorrectionConstant; | ||
| @Param | ||
| public Implementation implementation; |
| IOUtils.rm(path); | ||
| } | ||
| @Param | ||
| public Function function; |
...arks/src/main/java/org/elasticsearch/benchmark/vector/scorer/VectorScorerInt7uBenchmark.java
Show resolved
Hide resolved
| @Benchmark | ||
| public float squareDistanceNativeQuery() throws IOException { | ||
| return nativeSqrScorerQuery.score(1); | ||
| public float scoreQuery() throws IOException { |
There was a problem hiding this comment.
What happens if these are run on Java 21? (supportsHeapSegments() returns false?)
Is there a way to filter these benchmarks out?
There was a problem hiding this comment.
The method will throw an exception (NPE in this case), and the test will just be ignored by JMH
There was a problem hiding this comment.
A bit brutal :D but if it works...
.../src/main/java/org/elasticsearch/benchmark/vector/scorer/VectorScorerInt7uBulkBenchmark.java
Outdated
Show resolved
Hide resolved
ldematte
left a comment
There was a problem hiding this comment.
LGTM but I'd like for @ChrisHegarty to give it a look too.
I remember he was thinking at a naming convention or another way to run a subset of benchmarks -- JMH uses pattern matching on the function names, so being consisten pays.
Adding parameters for dimensions like implementation or function might help here, as they can be used as "filters" (e.g. just run all the benchmarks for native and dot product).
Also, do you have an example of what the output will look like?
Right. I think that the refactoring is probably ok, but I'd like to see how what the output looks like, and how we can more easily select subsets with various regex patterns. |
|
This command: produces this output: |
e9d4e7f to
bff2990
Compare
bff2990 to
348cd94
Compare
|
Thanks for the output. I think it is nice, as it gives us a way (using parameters) to filter/execute benchmarks over multiple dimensions (function, like you did in the example, but also implementation). |
This makes it a lot more scalable, so more functions can be added without adding more test methods.
So that the benchmark tests continue to work, the random data generation is pulled out into
VectorDataclasses that can be re-used across several different Benchmark classes for specific implementations, to ensure the results are still as expected.