Description

[Spinoff from #12306]

There have been many discussions and polls about what to do about the existing (weakly enforced) limit of aKNN vector dimensionality in Lucene.

This issue represents Option 3 in @alessandrobenedetti's recent poll thread.

Since it is this Codec component (currently Lucene95HnswVectorsFormat) that is implementing the HNSW approach for approximate KNN, it makes sense that it should be the one to enforce any limits (dimensionality, max connections, beam width, etc.). In fact, it already seems to enforce some limits -- I see MAXIMUM_BEAM_WIDTH = 3200 and MAXIMUM_MAX_CON = 512. Once we do this, users can still fork their own Codec to change limits, or implement a different aKNN algorithm, etc., and it will be clear that they are no longer using Lucene's default Codec so index format backwards compatibility is no longer ensured.

Version and environment details

No response