hmm as I work with this I am realizing this search interface is not really what I'm after ... basically I would like to be able to combine a DocIdSet from a Query with a FlatVectorScorer so that I can score the document using a quantized dot-product. So I think what I need is a RandomVectorScorer returned from the FlatVectorsReader and some way to go from docid->ord for my vector field. But it seems we only support ordToDoc now and not the other way?

@msokolov getByteVectorValues and getFloatVectorValues already do this. They provide an iterator that will return VectorScorer which can score and forward iterate via doc ids (supports advance, next, etc.).

Also, with the "flat codecs" it doesn't make much sense to gather up the accepted docs, place the results in a collector, etc. You want to score while you iterate. It would likely be way cheaper than doing it all up front like knn is doing.

Yes, of course, OK. So I think this will work like this if I leave the search() methods as throwing UnsupportedOperationException, then use PerFieldKnnVectorValues to tie a field to a Lucene99ScalarQuantizedVectorsFormat, at which point I can call getFloatVectorValues and use the returned scorer's score() method. Or perhaps call getQuantizedVectorValues directly and use its score method. I'll give it a try.

I'm OK with all that. If this becomes popular we might want to find a way that doesn't require using codec-specific classes, but we can let that sit for now I think. Thank you!

@msokolov I suppose that is ok for folks that understand the depths of codec formats. But we are seriously complicating somebody's life that wants to use kNN and scalar quantized flat codec. Now they cannot use the knn query, but instead have to use some specialized query they come up with themselves.

I think we at least should have a "flat vector query"? or something? Something that always does exact search so that typical users don't have to do this iterating & casting stuff?

@msokolov I suppose that is ok for folks that understand the depths of codec formats. But we are seriously complicating somebody's life that wants to use kNN and scalar quantized flat codec. Now they cannot use the knn query, but instead have to use some specialized query they come up with themselves.

+1 on this comment.

I think we at least should have a "flat vector query"? or something? Something that always does exact search so that typical users don't have to do this iterating & casting stuff?

Rather than having a new query can we just reuse the old query where for a field we get the right KNNVectorsReader and hit the search interface of the reader that does the exact search or HNSW based search based on whatever implementation the KNNVectorsReader have used.

I hear you all, this seems a little weird. One thing is ... you cannot get access to this stuff at all unless you are willing to create a custom Codec. So by definition we have weeded out people who don't want to mess around with Codecs. Another thing is ... we don't really want to create a trap for people who are trying to find top N vector-distance documents matching some constraint. They might think by blindly following the API trail/trap that they should create a BitSet representing their constraint and pass that in to a Knn*VectorQuery? When instead they would be better off executing their constraint as the primary Query and using the VectorValues API to do the scoring. I'm struggling to see the use case for this brute-force query; it just seems like a trap to me

Maybe we can make a KnnVectorScoreQuery that wraps another Query, supplying the vector score. Actually I think you can already do this with FunctionScoreQuery + a DoubleValuesSource wrapping a KnnVectorScorer? Perhaps we should provide that as a single Query?

So, doing this means the KnnFloatVectorQuery will throw an error if its used against a field with a codec using this reader.

Traditionally, query's were fairly ignorant of the underlying codec used to store the field.

It does seem weird to have a codec that the Knn queries could be executed against throw an exception.

I think either:

• The knn queries should automatically do the correct things (e.g. not eagerly rewrite themselves, etc.)
• We don't throw on search like this.

The knn queries should automatically do the correct things (e.g. not eagerly rewrite themselves, etc.)

I am not sure how to do this other than having something that says "Supports approximate search" to the leaf readers. Or a try{}catch() in the query, but then we are using exceptions as a query flow, which seems trappy and bad.

I did toy with adding a "supportsSearch" flag when this problem surfaced due to BaseKnnVectorsFormatTestCase. Potentially we could return no results. I feel like this is the moral equivalent of a stored Field with IndexOptions.NONE (Like StoredField.TYPE)? But we have tended to not want to make such vector-representation choices a part of the Fields API, so it looks different.

Since this is the case for all FlatVectorsReader objects, think we should put this override there?

Maybe this should be in FlatVectorsReader

I wonder why for Lucene99FlatVectorsFormat we allow an external name to be provided. Is this because it isn't really ever loaded via the SPI?

Seems like Lucene99FlatVectorsFormat shouldn't accept a name as a parameter and simply provide super with the correct name.

Makes sense. TBH I kind of just filled out the blanks with this one without thinking much about it, but I agree the name is really only useful for the SPI interface.

I wonder why a unique name here is required.

The top level format name is HnswBitVectorsFormat, and it doesn't really do anything to the inner format, it simply overrides the scoring mechanism.

on the subject of the brute-force query I started to look at adding one and found we already have o.a.l.queries.function.valuesource.VectorSimilarityFunction -- I think maybe all we need here is a static convenience method that produces a FunctionQuery wrapping one of these things although I'll confess I'm not entirely conversant with this API. It seems like it might want to be rewritten in terms of VectorScorer.