Lucene99 format uses the vector scorer supplier when building the HNSW graph - see https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java#L691. This references the byte/float heap vector directly, which means the native scorers in our supplier implementations can't be used, as they need a MemorySegment from on-disk vectors.

JDK22 supports creating MemorySegments from heap arrays, so we could create a JDK22 version of VectorScorerFactoryImpl that creates memory segments from the byte/float vector values, and uses native code on that.