Adds new `bit` element_type for dense_vectors by benwtrent · Pull Request #110059 · elastic/elasticsearch

benwtrent · 2024-06-21T17:33:51Z

This commit adds bit vector support by adding element_type: bit for vectors. This new element type works for indexed and non-indexed vectors. Additionally, it works with hnsw and flat index types. No quantization based codec works with this element type, this is consistent with byte vectors.

bit vectors accept up to 32768 dimensions in size and expect vectors that are being indexed to be encoded either as a hexidecimal string or a byte[] array where each element of the byte array represents 8 bits of the vector.

bit vectors support script usage and regular query usage. When indexed, all comparisons done are xor and popcount summations (aka, hamming distance), and the scores are transformed and normalized given the vector dimensions. Note, indexed bit vectors require l2_norm to be the similarity.

For scripts, l1norm is the same as hamming distance and l2norm is sqrt(l1norm). dotProduct and cosineSimilarity are not supported.

Note, the dimensions expected by this element_type are always to be divisible by 8, and the byte[] vectors provided for index must be have size dim/8 size, where each byte element represents 8 bits of the vectors.

closes: #48322

…or-support

elasticsearchmachine · 2024-06-21T17:34:14Z

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine · 2024-06-21T17:34:15Z

Hi @benwtrent, I've created a changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

benwtrent · 2024-06-21T17:46:31Z

I am planning on adding docs soonish

…or-support

docs/changelog/110059.yaml

...inless/src/yamlRestTest/resources/rest-api-spec/test/painless/146_dense_vector_bit_basic.yml

server/src/main/java/org/elasticsearch/index/codec/vectors/ES815BitFlatVectorsFormat.java

benwtrent · 2024-06-25T17:28:27Z

@mayya-sharipova just pushed:

I dropped support for cosine & dot product in the script
Adding docs about this and what l1, l2, and hamming are for bit vectors
Also, I noticed I didn't have tests & such for magnitude & bit vectors, added that and optimized the magnitude calculation.

docs/reference/vectors/vector-functions.asciidoc

benwtrent · 2024-06-26T00:47:41Z

@elasticmachine update branch

mayya-sharipova · 2024-06-26T13:23:07Z

@benwtrent Thanks, the code LGTM.

One thing that concerns me how we use l2_norm similarity parameter for defining bit vectors:

"bit_vector": {
    "type": "dense_vector",
    "dims" : 40,
    "element_type" : "bit",
    "similarity": "l2_norm"
}

because we don't actually use l2_norm, we use l1_norm or hamming for scoring calculations, but we don't have those similarity metrics defined in mapping.
Could it be confusing for users that we define "l2_norm", but never use in score calculations? What do you think?

One suggestion I have is NOT allow to provide any "similarity" metric for bit vectors in mappings for users, and say in documentation that "similarity" is not defined for bit vectors. What do you think of this suggestion?

benwtrent · 2024-06-26T14:34:13Z

@mayya-sharipova

One thing that concerns me how we use l2_norm similarity parameter for defining bit vectors:
...
because we don't actually use l2_norm, we use l1_norm or hamming for scoring calculations, but we don't have those similarity metrics defined in mapping.

So, our l2_norm calculation for all our other data types is just the squareDifference, we don't actually take the square root.

Lets take some bits for example: 1011 and 1001.

l2_norm would be (1-1)^2 + (0-0)^2 + (1-0)^2 + (1-1)^2 = 0 + 0 + 1 + 0 = 1
hamming is then 1^1 + 0^0 + 1^0 + 1^1 = 0 + 0 + 1 + 0 = 1

For bit vectors & how we already score things, l1_norm l2_norm and hamming are all the same.

benwtrent · 2024-06-26T14:35:14Z

@mayya-sharipova also, if users don't provide any similarity, we default to l2_norm for bit. So, its not strictly necessary at all.

mayya-sharipova · 2024-06-26T17:32:33Z

@benwtrent Thanks for the detailed explanation. I guess it is ok to keep l2_norm as defafult.

benwtrent · 2024-06-26T17:48:57Z

@elasticmachine update branch

…114407) **Description:** This PR addresses the issue described in [#114402](#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [#114402](#114402) - Introduced in [#110059](#110059)

…lastic#114407) **Description:** This PR addresses the issue described in [elastic#114402](elastic#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [elastic#114402](elastic#114402) - Introduced in [elastic#110059](elastic#110059)

…lastic#114407) **Description:** This PR addresses the issue described in [elastic#114402](elastic#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [elastic#114402](elastic#114402) - Introduced in [elastic#110059](elastic#110059) (cherry picked from commit 465c65c)

…114407) (#114756) **Description:** This PR addresses the issue described in [#114402](#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [#114402](#114402) - Introduced in [#110059](#110059) Co-authored-by: Rassyan <yjkhngds@gmail.com>

…lastic#114407) **Description:** This PR addresses the issue described in [elastic#114402](elastic#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [elastic#114402](elastic#114402) - Introduced in [elastic#110059](elastic#110059)

…114407) **Description:** This PR addresses the issue described in [#114402](#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [#114402](#114402) - Introduced in [#110059](#110059)

…ld (#114407) (#114759) * Fix Synthetic Source Handling for `bit` Type in `dense_vector` Field (#114407) **Description:** This PR addresses the issue described in [#114402](#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [#114402](#114402) - Introduced in [#110059](#110059) (cherry picked from commit 465c65c) * fixing backport of search capabilities * fixing license header * adding capabilities to RestSearchAction * fixing backport * spotless * muting teset for ccs * adding capabilities to the ccs test runner --------- Co-authored-by: Rassyan <yjkhngds@gmail.com>

…lastic#114407) **Description:** This PR addresses the issue described in [elastic#114402](elastic#114402), where the `synthetic_source` feature does not correctly handle the `bit` type in `dense_vector` fields when `index` is set to `false`. The root cause of the issue was that the `bit` type was not properly accounted for, leading to an array that is 8 times the size of the actual `dims` value of docvalue. This mismatch will causes an array out-of-bounds exception when reconstructing the document. **Changes:** - Adjusted the `synthetic_source` logic to correctly handle the `bit` type by ensuring the array size accounts for the 8x difference in dimensions. - Added yaml test to cover the `bit` type scenario in `dense_vector` fields with `index` set to `false`. **Related Issues:** - Closes [elastic#114402](elastic#114402) - Introduced in [elastic#110059](elastic#110059)

benwtrent added 11 commits June 18, 2024 16:22

Add binary vector support for HNSW and FLAT indices

1de8db4

iter

4dc9751

iter

4a4a189

iter

2dd2bc2

iter

008c897

Merge remote-tracking branch 'upstream/main' into feature/binary-vect…

f4cd592

…or-support

iter

e50f2cd

iter

5fa960e

iter

8ed1f6e

removed unnecessary change

239bc55

iter

5af7b40

benwtrent added >feature release highlight :Search Relevance/Vectors Vector search v8.15.0 labels Jun 21, 2024

benwtrent requested a review from a team June 21, 2024 17:33

benwtrent requested a review from a team as a code owner June 21, 2024 17:33

elasticsearchmachine added the Team:Search Meta label for search team label Jun 21, 2024

Update docs/changelog/110059.yaml

0e8d116

benwtrent added 3 commits June 24, 2024 08:09

Merge remote-tracking branch 'upstream/main' into feature/binary-vect…

0ff4f35

…or-support

fixing test

7d21e86

adding docs, fixing test

ccbc1ca

mayya-sharipova reviewed Jun 24, 2024

View reviewed changes

fixing tests

81fe98b

benwtrent requested a review from mayya-sharipova June 24, 2024 17:09

benwtrent added 2 commits June 24, 2024 13:32

fixing test

1c522a4

fixing moar tests

9367176

fixing tests

e3b07dd

mayya-sharipova reviewed Jun 25, 2024

View reviewed changes

docs/reference/vectors/vector-functions.asciidoc Show resolved Hide resolved

benwtrent requested a review from mayya-sharipova June 25, 2024 19:14

benwtrent and others added 2 commits June 25, 2024 15:15

fixing docs

8cbf12e

Update 110059.yaml

295a506

benwtrent added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Jun 26, 2024

Merge branch 'main' into feature/binary-vector-support

a32c39b

mayya-sharipova approved these changes Jun 26, 2024

View reviewed changes

benwtrent added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jun 26, 2024

Merge branch 'main' into feature/binary-vector-support

cc57e16

elasticsearchmachine merged commit 5add44d into elastic:main Jun 26, 2024

benwtrent deleted the feature/binary-vector-support branch June 26, 2024 18:49

Rassyan mentioned this pull request Oct 9, 2024

Fix Synthetic Source Handling for bit Type in dense_vector Field #114407

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds new `bit` element_type for dense_vectors#110059

Adds new `bit` element_type for dense_vectors#110059
elasticsearchmachine merged 28 commits intoelastic:mainfrom
benwtrent:feature/binary-vector-support

benwtrent commented Jun 21, 2024 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jun 21, 2024

Uh oh!

elasticsearchmachine commented Jun 21, 2024

Uh oh!

benwtrent commented Jun 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benwtrent commented Jun 25, 2024

Uh oh!

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

mayya-sharipova commented Jun 26, 2024 •

edited

Loading

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

mayya-sharipova commented Jun 26, 2024

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

benwtrent commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 21, 2024

Uh oh!

elasticsearchmachine commented Jun 21, 2024

Uh oh!

benwtrent commented Jun 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benwtrent commented Jun 25, 2024

Uh oh!

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

mayya-sharipova commented Jun 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

mayya-sharipova commented Jun 26, 2024

Uh oh!

benwtrent commented Jun 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

benwtrent commented Jun 21, 2024 •

edited

Loading

mayya-sharipova commented Jun 26, 2024 •

edited

Loading