Summary

Enable native vector scoring during HNSW graph construction by storing vectors off-heap in MemorySegments and providing direct native access to them, bypassing Lucene's opaque wrappers. This yields a reduction in indexing time.

Motivation

Previously, native vector scoring (via VectorScorerFactoryImpl) was only available for the search path, where vectors are memory-mapped from disk via MemorySegmentAccessInput. During indexing, HNSW graph
construction used Lucene's DefaultFlatVectorScorer with on-heap float[] arrays -- missing out on our optimized SIMD implementations.
The challenge: Lucene's Lucene99HnswVectorsWriter calls FloatVectorValues.fromFloats(flatFieldVectorsWriter.getVectors()), creating an anonymous wrapper that hides the underlying vector storage. We needed to
(a) control the storage, (b) make it off-heap so native code can access it, and (c) bridge through Lucene's opaque wrappers.

Approach

The work progressed through several layered steps:
Step 0: prepare FloatVectorScorerSupplier to work without MemorySegmentAccessInput -- Introduce a MemorySegmentAccessor abstraction in FloatVectorScorerSupplier to decouple it from MemorySegmentAccessInput, allowing native vector scoring for both memory-mapped and heap-resident data, with adapters for concrete cases. Update ES93FlatVectorScorer to attempt native scoring even when HasIndexSlice is not available (i.e., on-heap vectors during initial HNSW graph construction), and delegate raw float scoring to ES93FlatVectorScorer instead of Lucene's default scorer.
Step 1: Own the indexing path -- Rather than forking Lucene99FlatVectorsWriter entirely, we created ES93FlatVectorsWriter as a delegating wrapper that uses VarHandles to access the delegate's private
meta/vectorData IndexOutput fields. This lets us override addField() and flush() while keeping mergeOneField/mergeOneFieldToIndex delegated. A writer instance either does the indexing path or the
merge path, never both.
Step 2: Refactor BQ writer hierarchy -- Converted BinaryQuantizedVectorsFieldWriter from an interface to an abstract class extending FlatFieldVectorsWriter<float[]>, adding a createFieldWriter factory
method to ES818BinaryQuantizedVectorsWriter for subclass overrides (used by ES93BinaryQuantizedVectorsWriter).
Step 3: Off-heap vector storage -- Created OffHeapFloatVectorStore and OffHeapByteVectorStore using the MRJar pattern (src/main stubs, src/main21 MemorySegment-backed implementations in simdvec).
Each vector is individually allocated in a shared Arena. ES93FlatFieldVectorsWriter uses these stores, with in-place normalizeByMagnitudes operating directly on off-heap segments.
Step 4: Bridge off-heap store to scorer -- The key insight: getVectors() returns named AbstractList wrappers (WrappedNativeFloatVectors, WrappedNativeByteVectors in simdvec) that expose the underlying
store via getStore(). VectorScorerFactoryImpl uses reflection to look through Lucene's anonymous FloatVectorValues wrapper, find the captured list, and extract the OffHeapFloatVectorStore directly --
eliminating the round-trip of materializing float[] from off-heap just to wrap them back into MemorySegment.
Step 5: Shared MemorySegmentAccessor -- Extracted from FloatVectorScorerSupplier's nested interface to a top-level interface shared by both float and byte scorer suppliers. Refactored
ByteVectorScorerSupplier from MemorySegmentAccessInput to MemorySegmentAccessor, enabling the same three-tier resolution (IndexInput -> off-heap store -> heap fallback) for byte vectors.
Step 6: Native sparse bulk scoring -- Added vec_sqrf32_bulk_sparse, a native function taking f32_t* const* (array of pointers to individually-allocated vectors) instead of a contiguous block. This is the
"scatter" pattern: templatized sqrf32_inner_bulk on TData and a new mapper signature. The heap adapter can't use this path (JDK docs explicitly state MemorySegment.ofArray addresses can't populate a native
pointer struct), but OffHeapFloatStoreAdapter can because its vectors are truly off-heap. MemorySegmentAccessor.segmentForEntriesOrNull(int[], Arena) builds the pointer array; the Arena parameter is
caller-managed for clean lifecycle control.

Design decisions

• Delegation + VarHandle over full copy: We initially considered forking Lucene99FlatVectorsWriter entirely, but the VarHandle approach is less code and easier to maintain across Lucene upgrades -- we only
  own the indexing path.
• Reflection bridge over Lucene fork: Using reflection to unwrap Lucene's anonymous FloatVectorValues is pragmatic. A cleaner long-term solution (e.g., custom HnswVectorsWriter) may replace it later.
• MRJar for MemorySegment APIs: java.lang.foreign is preview in JDK 21, finalized in JDK 22. OffHeapFloatVectorStore, OffHeapByteVectorStore, and ArenaUtil use src/main stubs +
  src/main21/src/main22 implementations.
• Individual segment allocation: Each vector gets its own MemorySegment (vs one large contiguous segment) because vectors arrive one at a time during indexing. This naturally leads to the sparse bulk scoring
  approach.

Benchmark results

GIST-1M, 200k docs, float32, euclidean, HNSW(m=16, efC=200), BQ:

• Before (heap, non-native scoring): 225s indexing
• After (off-heap, native sparse bulk scoring): 162s indexing
• Speedup: ~28%

Known limitations

• Sparse bulk scoring currently implemented only for Euclidean distance; DotProduct and MaxInnerProduct fall back to per-pair scoring.
• Reflection-based store extraction is a pragmatic bridge; a cleaner approach may replace it later.
• Further work may be needed to address other implementations (e.g., for Int7SQVectorScorerSupplier -- but we need to check if we need it).

Test plan

• All libs:simdvec tests pass
• All libs:native tests pass
• ES93 vector format tests pass (updated testToString expectations)
• Verified via async-profiler flamegraph that native sparse bulk scoring path is used during HNSW graph construction
• Benchmarked with KnnIndexTester on GIST-1M (gives a ~30% indexing speedup)

Co-authored with Cursor