Vector model - implementation preview by lvca · Pull Request #1148 · ArcadeData/arcadedb

lvca · 2023-06-23T16:43:43Z

This is the first piece of the Vector Model for ArcadeDB. We extended the amazing Open Source HNSW library by Jelmer Kuperus (https://github.com/jelmerk/hnswlib) to be supported in ArcadeDB as an index.

HNSW stands for "Hierarchical Navigable Small World" and it's probably the best algorithm to index multi-dimensional data. For more information look at the paper "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs".

After trying to load some of the most popular datasets, we used the publicly available Word2Vec with 2M words, 300 dimenions.

To experiment this new feature, just run the Java class com.arcadedb.index.vector.FastTextDatabase that downloads the first time the dataset, and start indexing all the words in ArcadeDB using the new HNSW index.

The best way to load vectors in ArcadeDB is by using a RAM-based index to load the dataset quickly:

HnswVectorIndexRAM<String, float[], Word, Float> hnswIndex = HnswVectorIndexRAM.newBuilder(300, DistanceFunctions.FLOAT_INNER_PRODUCT, words.size())
          .withM(16).withEf(200).withEfConstruction(200).build();

And then make it persistent in ArcadeDB:

persistentIndex = hnswIndex.createPersistentIndex(database)//
.withVertexType("Word").withEdgeType("Proximity").withVectorPropertyName("vector").withIdProperty("name").create();

persistentIndex.save();

With the code above, you're told to store the HNSW index inside "Word" vertices by using the edge type "Proximity". The vector property will be stored in the property "vector", as a float[300]. The actual word as a string is saved in the property "name".

The types and properties mentioned above are automatically created if not already in the database.

After the first import, your data is in ArcadeDB, so you can just open your database and get the HNSW index as a normal index:

persistentIndex = (HnswVectorIndex) database.getSchema().getIndexByName("Word[name,vector]");

Note the index name is composed of the selected vertex type and the name and vector properties.

The Java test class pulls a random word from the database and looks for its neighbors.

List<SearchResult<Vertex, Float>> approximateResults = persistentIndex.findNeighbors(input, k);

Where input is the word and K is the maximum number of neighbors to retrieve, ordered by the highest proximity, namely the K closest words will be returned.

The result is an ordered list where you can get the word vertex and its proximity (as a Float).

Fixed issue 1102

Fixed issue #1097

… index

…timized for RAM implementation

They are not stored as List with dynamic type for each item, but rather as array of element of the same type. This saves much more space and it is much faster because avoid java autoboxing by using literals

…o see the timestamp

lvca · 2023-06-23T21:04:15Z

Adding some other information about the implementation we chose. Since the HNSW index is based on a graph, why reimplement the wheel in ArcadeDB which is a native Graph Database? Also, I'd like to expose all the settings HNSW implementation has, so the user can tune the index for performance and/or accuracy of the results.

So we store the vectors as arrays (compressed) inside the user's vertex type. For proximity, the user can specify the edge type to use. In the end, the raw vector embeddings are stored in the vertices and the proximity concept is an actual edge. This allows to reuse of the whole Graph Database and allows the user to execute queries against embeddings and proximity built by the HNSW index.

So the HNSW index is not stored in its own files (like with the LSMTree index), but rather enhances the user domain.

lvca · 2023-06-23T21:08:44Z

The things still missing here are:

extend ArcadeDB Importer to import vectors directly into the database without writing code
provide new SQL functions, like findNeighbors(<key>, <K>) to allow the user to use the HNSW index without writing code
a lot of test cases still missing
documentation still totally missing (@gramian is already working on this)

hrstoyanov · 2023-06-28T19:30:11Z

@lvca This looks impressive on first review! I will try to dig deeper, as time permits... Thank you!

I tried to build it of the branch, but got this:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project arcadedb-engine: Compilation failure: Compilation failure: 
[ERROR] /workspaces/arcadedb/engine/src/main/java/com/arcadedb/index/vector/DistanceFunctionFactory.java:[45,68] cannot find symbol
[ERROR]   symbol:   variable FLOAT_CHEBYSHEV_DISTANCE
[ERROR]   location: class com.github.jelmerk.knn.DistanceFunctions
[ERROR] /workspaces/arcadedb/engine/src/main/java/com/arcadedb/index/vector/DistanceFunctionFactory.java:[53,69] cannot find symbol
[ERROR]   symbol:   variable DOUBLE_CHEBYSHEV_DISTANCE
[ERROR]   location: class com.github.jelmerk.knn.DistanceFunctions

I also have a couple of questions:

why not byte[] added?
where can I see more of the "compression" mechanism? Does this involve "quantization" -> reducing Java native types with procession lose?

On my "dream list" for the future:

I guess this is more for @jelmerk and hnslib project, but in future it would be awesome to vectorize the distance computations
to use Java 21's vector API . This can easily speed things up 15X ...10X or more, on both X84 and ARM.
maybe use native arrays with size that can be bigger than Integer.MAX_INTEGER, allocated of-the-heap (the new memory segments) as in the new FFI&MAPI

lvca · 2023-06-28T20:15:24Z

I tried to build it of the branch, but got this:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project arcadedb-engine: Compilation failure: Compilation failure: 
[ERROR] /workspaces/arcadedb/engine/src/main/java/com/arcadedb/index/vector/DistanceFunctionFactory.java:[45,68] cannot find symbol
[ERROR]   symbol:   variable FLOAT_CHEBYSHEV_DISTANCE
[ERROR]   location: class com.github.jelmerk.knn.DistanceFunctions
[ERROR] /workspaces/arcadedb/engine/src/main/java/com/arcadedb/index/vector/DistanceFunctionFactory.java:[53,69] cannot find symbol
[ERROR]   symbol:   variable DOUBLE_CHEBYSHEV_DISTANCE
[ERROR]   location: class com.github.jelmerk.knn.DistanceFunctions

Fixed on this PR. The issue was that @gramian created such functions on the forked HNSW project, but it's not on maven. so it won't compile it. I moved those new functions under ArcadeDB project.

I also have a couple of questions:

why not byte[] added?

Because we already have the BINARY type that is a byte[]. Also, bytes can't be compressed as we do with other types, because the minimum number occupies a byte.

where can I see more of the "compression" mechanism? Does this involve "quantization" -> reducing Java native types with procession lose?

If you write an external quantization, maybe you can use short[] and the index already supports it. @hrstoyanov do you know any OS Java quantization libraries we could embed? The poor man quantization in my mind is to enum all the values and then create a map of them with the most used in the 1st place, and so on. So we could store only the index from the map that could occupy from 1 byte up. I noticed with Word2Vac that the range of values (as floats) is very limited, so it would save some space.

On my "dream list" for the future:

I guess this is more for @jelmerk and hnslib project, but in future it would be awesome to vectorize the distance computations
to use Java 21's vector API . This can easily speed things up 15X ...10X or more, on both X84 and ARM.

Definitely. We're going to provide such implementation in ArcadeDB, no matter what the original project does.

maybe use native arrays with sizes that can be bigger than Integer.MAX_INTEGER, allocated of-the-heap (the new memory segments) as in the new FFI&MAPI

This can be useful in the load part, where we build the index, but once is built, the HNSW index is not loaded in RAM, only lazy loaded the piece of the graph requested for the query.

Applied the PR from jelmerk/hnswlib#54

lvca · 2023-06-29T14:18:07Z

Inspired by the Qdrant Benchmark page (https://qdrant.tech/benchmarks/) I downloaded the GloVe-100-Angular dataset (100 dimensions, 1.2M entries) and loaded it into ArcadeDB. Below are some numbers running on my Mac Book Pro 2019 (Intel CPU, 32GB RAM)

The dataset was loaded and indexed in ArcadeDB in 9 minutes using only 8GB RAM.
The RPS (request per second) is 435.58 by using M=16 and EF=128

I can't see what's the accuracy of the responses, I guess it should be measured against a distance calculated with all the vector embeddings.

#995

lvca · 2023-06-30T06:32:33Z

Just integrated the vector index into the Importer. Now you can import a Word2Vec or GloVe text file with the IMPORT DATABASE:

import database file://glove-angular-100.txt
with distanceFunction = 'cosine', m = 16, ef = 128, efConstruction = 128, 
vertexType = 'Word', edgeType = 'Proximity', vectorProperty = 'vector', idProperty = 'name'"

…ome refactoring

codacy-production · 2023-07-03T17:47:14Z

Coverage summary from Codacy

Merging #1148 (58d0f55) into main (5364986) - See PR on Codacy

Coverage variation	Diff coverage
Report missing for `5364986`¹	24.16%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`5364986`)	Report Missing	Report Missing	Report Missing
Head commit (`58d0f55`)	52743	32733	62.06%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1148)	1759	425	24.16%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

Bumps `jline.version` from 3.28.0 to 3.29.0. Updates `org.jline:jline-terminal` from 3.28.0 to 3.29.0 Release notes *Sourced from [org.jline:jline-terminal's releases](https://github.com/jline/jline3/releases).* > 3.29.0 > ------ > > 💥 Breaking changes > ----------------------- > > * Undeprecate ConsolePrompt methods et al ([#1148](https://redirect.github.com/jline/jline3/pull/1148)) [`@quintesse`](https://github.com/quintesse) > > 🚀 New features and improvements > ------------------------------- > > * Add text-only prompt element for console-ui ([#1138](https://redirect.github.com/jline/jline3/pull/1138)) [`@quintesse`](https://github.com/quintesse) > * Dynamic console-ui prompt improvements, see [#1051](https://redirect.github.com/jline/jline3/issues/1051) ([#1132](https://redirect.github.com/jline/jline3/pull/1132)) [`@quintesse`](https://github.com/quintesse) > > 🐛 Bug Fixes > ----------- > > * AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/pull/1161)) [`@cstamas`](https://github.com/cstamas) > * Add overloaded no-arg compile method ([#1137](https://redirect.github.com/jline/jline3/pull/1137)) [`@iflan`](https://github.com/iflan) > * Move catch to proper place ([#1133](https://redirect.github.com/jline/jline3/pull/1133)) [`@cstamas`](https://github.com/cstamas) > > 📦 Dependency updates > -------------------- > > * Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/pull/1163)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/pull/1159)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/pull/1155)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.sonatype.central:central-publishing-maven-plugin from 0.6.0 to 0.7.0 ([#1149](https://redirect.github.com/jline/jline3/pull/1149)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/pull/1147)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/pull/1146)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.apache.ivy:ivy from 2.5.2 to 2.5.3 ([#1144](https://redirect.github.com/jline/jline3/pull/1144)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump net.java.dev.jna:jna from 5.15.0 to 5.16.0 ([#1142](https://redirect.github.com/jline/jline3/pull/1142)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump junit.version from 5.11.3 to 5.11.4 ([#1135](https://redirect.github.com/jline/jline3/pull/1135)) @[dependabot[bot]](https://github.com/apps/dependabot) > > 👻 Maintenance > ------------- > > * Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/pull/1158)) [`@snazy`](https://github.com/snazy) Commits * [`c095db8`](jline/jline3@c095db8) [maven-release-plugin] prepare release jline-3.29.0 * [`a6a7786`](jline/jline3@a6a7786) Undeprecate ConsolePrompt methods ([#1148](https://redirect.github.com/jline/jline3/issues/1148)) * [`ee5ac1b`](jline/jline3@ee5ac1b) Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/issues/1163)) * [`57030d4`](jline/jline3@57030d4) Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/issues/1158)) * [`f869217`](jline/jline3@f869217) AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/issues/1161)) * [`c7f7fbe`](jline/jline3@c7f7fbe) Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/issues/1159)) * [`ddfb352`](jline/jline3@ddfb352) Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/issues/1155)) * [`4504b46`](jline/jline3@4504b46) Bump org.sonatype.central:central-publishing-maven-plugin ([#1149](https://redirect.github.com/jline/jline3/issues/1149)) * [`5e6aa3e`](jline/jline3@5e6aa3e) Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/issues/1147)) * [`1e70700`](jline/jline3@1e70700) Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/issues/1146)) * Additional commits viewable in [compare view](jline/jline3@jline-3.28.0...jline-3.29.0) Updates `org.jline:jline-reader` from 3.28.0 to 3.29.0 Release notes *Sourced from [org.jline:jline-reader's releases](https://github.com/jline/jline3/releases).* > 3.29.0 > ------ > > 💥 Breaking changes > ----------------------- > > * Undeprecate ConsolePrompt methods et al ([#1148](https://redirect.github.com/jline/jline3/pull/1148)) [`@quintesse`](https://github.com/quintesse) > > 🚀 New features and improvements > ------------------------------- > > * Add text-only prompt element for console-ui ([#1138](https://redirect.github.com/jline/jline3/pull/1138)) [`@quintesse`](https://github.com/quintesse) > * Dynamic console-ui prompt improvements, see [#1051](https://redirect.github.com/jline/jline3/issues/1051) ([#1132](https://redirect.github.com/jline/jline3/pull/1132)) [`@quintesse`](https://github.com/quintesse) > > 🐛 Bug Fixes > ----------- > > * AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/pull/1161)) [`@cstamas`](https://github.com/cstamas) > * Add overloaded no-arg compile method ([#1137](https://redirect.github.com/jline/jline3/pull/1137)) [`@iflan`](https://github.com/iflan) > * Move catch to proper place ([#1133](https://redirect.github.com/jline/jline3/pull/1133)) [`@cstamas`](https://github.com/cstamas) > > 📦 Dependency updates > -------------------- > > * Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/pull/1163)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/pull/1159)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/pull/1155)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.sonatype.central:central-publishing-maven-plugin from 0.6.0 to 0.7.0 ([#1149](https://redirect.github.com/jline/jline3/pull/1149)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/pull/1147)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/pull/1146)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.apache.ivy:ivy from 2.5.2 to 2.5.3 ([#1144](https://redirect.github.com/jline/jline3/pull/1144)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump net.java.dev.jna:jna from 5.15.0 to 5.16.0 ([#1142](https://redirect.github.com/jline/jline3/pull/1142)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump junit.version from 5.11.3 to 5.11.4 ([#1135](https://redirect.github.com/jline/jline3/pull/1135)) @[dependabot[bot]](https://github.com/apps/dependabot) > > 👻 Maintenance > ------------- > > * Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/pull/1158)) [`@snazy`](https://github.com/snazy) Commits * [`c095db8`](jline/jline3@c095db8) [maven-release-plugin] prepare release jline-3.29.0 * [`a6a7786`](jline/jline3@a6a7786) Undeprecate ConsolePrompt methods ([#1148](https://redirect.github.com/jline/jline3/issues/1148)) * [`ee5ac1b`](jline/jline3@ee5ac1b) Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/issues/1163)) * [`57030d4`](jline/jline3@57030d4) Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/issues/1158)) * [`f869217`](jline/jline3@f869217) AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/issues/1161)) * [`c7f7fbe`](jline/jline3@c7f7fbe) Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/issues/1159)) * [`ddfb352`](jline/jline3@ddfb352) Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/issues/1155)) * [`4504b46`](jline/jline3@4504b46) Bump org.sonatype.central:central-publishing-maven-plugin ([#1149](https://redirect.github.com/jline/jline3/issues/1149)) * [`5e6aa3e`](jline/jline3@5e6aa3e) Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/issues/1147)) * [`1e70700`](jline/jline3@1e70700) Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/issues/1146)) * Additional commits viewable in [compare view](jline/jline3@jline-3.28.0...jline-3.29.0) Updates `org.jline:jline-terminal-jni` from 3.28.0 to 3.29.0 Release notes *Sourced from [org.jline:jline-terminal-jni's releases](https://github.com/jline/jline3/releases).* > 3.29.0 > ------ > > 💥 Breaking changes > ----------------------- > > * Undeprecate ConsolePrompt methods et al ([#1148](https://redirect.github.com/jline/jline3/pull/1148)) [`@quintesse`](https://github.com/quintesse) > > 🚀 New features and improvements > ------------------------------- > > * Add text-only prompt element for console-ui ([#1138](https://redirect.github.com/jline/jline3/pull/1138)) [`@quintesse`](https://github.com/quintesse) > * Dynamic console-ui prompt improvements, see [#1051](https://redirect.github.com/jline/jline3/issues/1051) ([#1132](https://redirect.github.com/jline/jline3/pull/1132)) [`@quintesse`](https://github.com/quintesse) > > 🐛 Bug Fixes > ----------- > > * AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/pull/1161)) [`@cstamas`](https://github.com/cstamas) > * Add overloaded no-arg compile method ([#1137](https://redirect.github.com/jline/jline3/pull/1137)) [`@iflan`](https://github.com/iflan) > * Move catch to proper place ([#1133](https://redirect.github.com/jline/jline3/pull/1133)) [`@cstamas`](https://github.com/cstamas) > > 📦 Dependency updates > -------------------- > > * Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/pull/1163)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/pull/1159)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/pull/1155)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.sonatype.central:central-publishing-maven-plugin from 0.6.0 to 0.7.0 ([#1149](https://redirect.github.com/jline/jline3/pull/1149)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/pull/1147)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/pull/1146)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump org.apache.ivy:ivy from 2.5.2 to 2.5.3 ([#1144](https://redirect.github.com/jline/jline3/pull/1144)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump net.java.dev.jna:jna from 5.15.0 to 5.16.0 ([#1142](https://redirect.github.com/jline/jline3/pull/1142)) @[dependabot[bot]](https://github.com/apps/dependabot) > * Bump junit.version from 5.11.3 to 5.11.4 ([#1135](https://redirect.github.com/jline/jline3/pull/1135)) @[dependabot[bot]](https://github.com/apps/dependabot) > > 👻 Maintenance > ------------- > > * Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/pull/1158)) [`@snazy`](https://github.com/snazy) Commits * [`c095db8`](jline/jline3@c095db8) [maven-release-plugin] prepare release jline-3.29.0 * [`a6a7786`](jline/jline3@a6a7786) Undeprecate ConsolePrompt methods ([#1148](https://redirect.github.com/jline/jline3/issues/1148)) * [`ee5ac1b`](jline/jline3@ee5ac1b) Bump groovy.version from 4.0.24 to 4.0.25 ([#1163](https://redirect.github.com/jline/jline3/issues/1163)) * [`57030d4`](jline/jline3@57030d4) Replace `Path.toFile()` ([#1158](https://redirect.github.com/jline/jline3/issues/1158)) * [`f869217`](jline/jline3@f869217) AnsiConsole should always obey the terminal (fixes [#1160](https://redirect.github.com/jline/jline3/issues/1160)) ([#1161](https://redirect.github.com/jline/jline3/issues/1161)) * [`c7f7fbe`](jline/jline3@c7f7fbe) Bump org.graalvm.sdk:graal-sdk from 24.1.1 to 24.1.2 ([#1159](https://redirect.github.com/jline/jline3/issues/1159)) * [`ddfb352`](jline/jline3@ddfb352) Bump com.diffplug.spotless:spotless-maven-plugin from 2.44.1 to 2.44.2 ([#1155](https://redirect.github.com/jline/jline3/issues/1155)) * [`4504b46`](jline/jline3@4504b46) Bump org.sonatype.central:central-publishing-maven-plugin ([#1149](https://redirect.github.com/jline/jline3/issues/1149)) * [`5e6aa3e`](jline/jline3@5e6aa3e) Bump com.diffplug.spotless:spotless-maven-plugin from 2.43.0 to 2.44.1 ([#1147](https://redirect.github.com/jline/jline3/issues/1147)) * [`1e70700`](jline/jline3@1e70700) Bump org.codehaus.gmavenplus:gmavenplus-plugin from 4.0.1 to 4.1.1 ([#1146](https://redirect.github.com/jline/jline3/issues/1146)) * Additional commits viewable in [compare view](jline/jline3@jline-3.28.0...jline-3.29.0) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

… [skip ci] Bumps [testcontainers](https://github.com/testcontainers/testcontainers-node) from 11.6.0 to 11.7.1. Release notes *Sourced from [testcontainers's releases](https://github.com/testcontainers/testcontainers-node/releases).* > v11.7.1 > ------- > > Changes > ------- > > 🐛 Bug Fixes > ----------- > > * Return empty registry credentials when none found [`@cristianrgreco`](https://github.com/cristianrgreco) ([#1153](https://redirect.github.com/testcontainers/testcontainers-node/issues/1153)) > > v11.7.0 > ------- > > Changes > ------- > > 🚀 Features > ---------- > > * Bump ryuk image version to 0.14 [`@digital88`](https://github.com/digital88) ([#1148](https://redirect.github.com/testcontainers/testcontainers-node/issues/1148)) > * Add CouchDB module [`@botflux`](https://github.com/botflux) ([#1136](https://redirect.github.com/testcontainers/testcontainers-node/issues/1136)) > > 🐛 Bug Fixes > ----------- > > * Fix MongoDB container excessive CPU usage [`@digital88`](https://github.com/digital88) ([#1146](https://redirect.github.com/testcontainers/testcontainers-node/issues/1146)) > > 📦 Dependency Updates > -------------------- > > * Bump mkdocs-material from 9.6.19 to 9.6.20 in the dependencies group @[dependabot[bot]](https://github.com/apps/dependabot) ([#1142](https://redirect.github.com/testcontainers/testcontainers-node/issues/1142)) > * Bump the dependencies group with 15 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#1141](https://redirect.github.com/testcontainers/testcontainers-node/issues/1141)) Commits * [`c243f8e`](testcontainers/testcontainers-node@c243f8e) Return empty registry credentials when none found ([#1153](https://redirect.github.com/testcontainers/testcontainers-node/issues/1153)) * [`8d320dc`](testcontainers/testcontainers-node@8d320dc) v11.7.0 * [`e5f28f8`](testcontainers/testcontainers-node@e5f28f8) Bump mkdocs-material from 9.6.19 to 9.6.20 in the dependencies group ([#1142](https://redirect.github.com/testcontainers/testcontainers-node/issues/1142)) * [`e7399ee`](testcontainers/testcontainers-node@e7399ee) Bump the dependencies group with 15 updates ([#1141](https://redirect.github.com/testcontainers/testcontainers-node/issues/1141)) * [`9c68c18`](testcontainers/testcontainers-node@9c68c18) Bump ryuk to 0.14 ([#1148](https://redirect.github.com/testcontainers/testcontainers-node/issues/1148)) * [`dc79944`](testcontainers/testcontainers-node@dc79944) Fix MongoDB container excessive CPU usage ([#1146](https://redirect.github.com/testcontainers/testcontainers-node/issues/1146)) * [`9271e97`](testcontainers/testcontainers-node@9271e97) Add CouchDB module ([#1136](https://redirect.github.com/testcontainers/testcontainers-node/issues/1136)) * [`885e117`](testcontainers/testcontainers-node@885e117) v11.6.0 * See full diff in [compare view](testcontainers/testcontainers-node@v11.6.0...v11.7.1) [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility\_score?dependency-name=testcontainers&package-manager=npm\_and\_yarn&previous-version=11.6.0&new-version=11.7.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

…udio [skip ci] [//]: # (dependabot-start) ⚠️ \*\*Dependabot is rebasing this PR\*\* ⚠️ Rebasing might not happen immediately, so don't worry if this takes some time. Note: if you make any changes to this PR yourself, they will take precedence over the rebase. --- [//]: # (dependabot-end) Bumps [testcontainers](https://github.com/testcontainers/testcontainers-node) from 11.2.1 to 11.7.1. Release notes *Sourced from [testcontainers's releases](https://github.com/testcontainers/testcontainers-node/releases).* > v11.7.1 > ------- > > Changes > ------- > > 🐛 Bug Fixes > ----------- > > * Return empty registry credentials when none found [`@cristianrgreco`](https://github.com/cristianrgreco) ([#1153](https://redirect.github.com/testcontainers/testcontainers-node/issues/1153)) > > v11.7.0 > ------- > > Changes > ------- > > 🚀 Features > ---------- > > * Bump ryuk image version to 0.14 [`@digital88`](https://github.com/digital88) ([#1148](https://redirect.github.com/testcontainers/testcontainers-node/issues/1148)) > * Add CouchDB module [`@botflux`](https://github.com/botflux) ([#1136](https://redirect.github.com/testcontainers/testcontainers-node/issues/1136)) > > 🐛 Bug Fixes > ----------- > > * Fix MongoDB container excessive CPU usage [`@digital88`](https://github.com/digital88) ([#1146](https://redirect.github.com/testcontainers/testcontainers-node/issues/1146)) > > 📦 Dependency Updates > -------------------- > > * Bump mkdocs-material from 9.6.19 to 9.6.20 in the dependencies group @[dependabot[bot]](https://github.com/apps/dependabot) ([#1142](https://redirect.github.com/testcontainers/testcontainers-node/issues/1142)) > * Bump the dependencies group with 15 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#1141](https://redirect.github.com/testcontainers/testcontainers-node/issues/1141)) > > v11.6.0 > ------- > > Changes > ------- > > 🚀 Features > ---------- > > * Skip checking registry on credential helper list [`@karol-bochenski`](https://github.com/karol-bochenski) ([#1121](https://redirect.github.com/testcontainers/testcontainers-node/issues/1121)) > * Add username option to Valkey container [`@rqbazan`](https://github.com/rqbazan) ([#1117](https://redirect.github.com/testcontainers/testcontainers-node/issues/1117)) > > 📖 Documentation > --------------- > > * Fix typo in usage example (RedisContainer) [`@lightningspirit`](https://github.com/lightningspirit) ([#1132](https://redirect.github.com/testcontainers/testcontainers-node/issues/1132)) > > 📦 Dependency Updates > -------------------- > > * Bump the dependencies group across 1 directory with 22 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#1137](https://redirect.github.com/testcontainers/testcontainers-node/issues/1137)) > * Bump the dependencies group across 1 directory with 15 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#1123](https://redirect.github.com/testcontainers/testcontainers-node/issues/1123)) > * Bump mkdocs-material from 9.6.17 to 9.6.18 in the dependencies group @[dependabot[bot]](https://github.com/apps/dependabot) ([#1118](https://redirect.github.com/testcontainers/testcontainers-node/issues/1118)) > * Bump mkdocs-material from 9.6.16 to 9.6.17 in the dependencies group @[dependabot[bot]](https://github.com/apps/dependabot) ([#1110](https://redirect.github.com/testcontainers/testcontainers-node/issues/1110)) > * Bump the dependencies group across 1 directory with 16 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#1111](https://redirect.github.com/testcontainers/testcontainers-node/issues/1111)) > * Bump actions/checkout from 4 to 5 in the actions group @[dependabot[bot]](https://github.com/apps/dependabot) ([#1112](https://redirect.github.com/testcontainers/testcontainers-node/issues/1112)) > * Remove msw override [`@cristianrgreco`](https://github.com/cristianrgreco) ([#1114](https://redirect.github.com/testcontainers/testcontainers-node/issues/1114)) > > v11.5.1 > ------- > > Changes > ------- > > 🐛 Bug Fixes > ----------- ... (truncated) Commits * [`c243f8e`](testcontainers/testcontainers-node@c243f8e) Return empty registry credentials when none found ([#1153](https://redirect.github.com/testcontainers/testcontainers-node/issues/1153)) * [`8d320dc`](testcontainers/testcontainers-node@8d320dc) v11.7.0 * [`e5f28f8`](testcontainers/testcontainers-node@e5f28f8) Bump mkdocs-material from 9.6.19 to 9.6.20 in the dependencies group ([#1142](https://redirect.github.com/testcontainers/testcontainers-node/issues/1142)) * [`e7399ee`](testcontainers/testcontainers-node@e7399ee) Bump the dependencies group with 15 updates ([#1141](https://redirect.github.com/testcontainers/testcontainers-node/issues/1141)) * [`9c68c18`](testcontainers/testcontainers-node@9c68c18) Bump ryuk to 0.14 ([#1148](https://redirect.github.com/testcontainers/testcontainers-node/issues/1148)) * [`dc79944`](testcontainers/testcontainers-node@dc79944) Fix MongoDB container excessive CPU usage ([#1146](https://redirect.github.com/testcontainers/testcontainers-node/issues/1146)) * [`9271e97`](testcontainers/testcontainers-node@9271e97) Add CouchDB module ([#1136](https://redirect.github.com/testcontainers/testcontainers-node/issues/1136)) * [`885e117`](testcontainers/testcontainers-node@885e117) v11.6.0 * [`ab7fbdd`](testcontainers/testcontainers-node@ab7fbdd) Bump mkdocs-material from 9.6.18 to 9.6.19 in the dependencies group ([#1128](https://redirect.github.com/testcontainers/testcontainers-node/issues/1128)) * [`785e1b2`](testcontainers/testcontainers-node@785e1b2) Fix typo in usage example (RedisContainer) ([#1132](https://redirect.github.com/testcontainers/testcontainers-node/issues/1132)) * Additional commits viewable in [compare view](testcontainers/testcontainers-node@v11.2.1...v11.7.1) [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility\_score?dependency-name=testcontainers&package-manager=npm\_and\_yarn&previous-version=11.2.1&new-version=11.7.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

lvca added 30 commits May 26, 2023 01:28

feat: refactoring of schema api using a builder for types

bda89d7

Fixed issue 1102

Merge branch 'main' into vector-model

85ef3bc

fix: SQLSCRIPT commit command with retry

bc67715

Fixed issue #1097

Merge branch 'main' into vector-model

6fe5d0c

Merge branch 'main' into vector-model

bc4ab9f

Merge branch 'main' into vector-model

dfb0f56

Merge branch 'main' into vector-model

79a6b9d

First version of VectorBuilder to hold parameters to build the vector…

0ca43d4

… index

First implementation (not tested yet) of HnswIndex derived from an op…

e6e638f

…timized for RAM implementation

chore: removed unused properties

f693bd9

feat: optimized support for arrays of numbers

56ce0a3

They are not stored as List with dynamic type for each item, but rather as array of element of the same type. This saves much more space and it is much faster because avoid java autoboxing by using literals

feat: first working version of importer + index.

b9ab2d6

feat: completed 1st impl of RAM+Database HSNW Index

82e067b

feat: vector test now select a random word

0951c06

Merge branch 'main' into vector-model

86275b5

test: vector test run, removed delay and converted output into logs t…

6bf2372

…o see the timestamp

Merge branch 'main' into vector-model

45b44e2

Merge branch 'main' into vector-model

d0780a3

Merge branch 'main' into vector-model

a3e8fce

First draft of create index command for vectors

a79ee05

Merge branch 'main' into vector-model

fa87176

Merge branch 'main' into vector-model

f89154f

Merge branch 'main' into vector-model

7ebe6f9

feat: implemented ArcadeDB's Index interface on HnswVectorIndex

7ff1ef4

feat: major refactoring to support vector indexes

809868a

Merge branch 'main' into vector-model

f170db6

test: removed initial count of words

2c38682

Merge branch 'main' into vector-model

d5d8833

Fixed compilation issue

6aac7f1

Merge branch 'main' into vector-model

ab759c2

Merge branch 'main' into vector-model

b86085e

Moved ChebyshevDistance into ArcadeDB project

1238cd8

lvca added 2 commits June 28, 2023 16:16

fix: hnsw -> init level connections under the control of maxM0 and maxM

18fcfbd

Applied the PR from jelmerk/hnswlib#54

feat: added statistics to the vector demo

e65845c

lvca added 2 commits June 30, 2023 02:29

integrated Word2Vec and GloVe format into the importer

44df542

#995

Merge branch 'main' into vector-model

0b6e4c3

lvca added 9 commits June 30, 2023 12:04

fix: automatically saved the vector index upon creation

6798497

feat: sql -> added vectorNeighbors() function

fa4d6d2

Merge branch 'main' into vector-model

3a89de5

fix: compilation warning

58ae355

feat: supported arrays in Types + auto create type and properties + s…

22447de

…ome refactoring

chore: refactoring with names

d98f6f7

chore: fixed typos

25db346

test: fixed broken tests

8d7c7da

test: fixed broken test

58d0f55

lvca merged commit 3dabad0 into main Jul 3, 2023

lvca deleted the vector-model branch July 3, 2023 19:03

lvca restored the vector-model branch July 3, 2023 19:03

lvca deleted the vector-model branch July 3, 2023 19:03

lvca restored the vector-model branch July 3, 2023 19:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Vector model - implementation preview#1148

Vector model - implementation preview#1148
lvca merged 50 commits intomainfrom
vector-model

lvca commented Jun 23, 2023

Uh oh!

lvca commented Jun 23, 2023 •

edited

Loading

Uh oh!

lvca commented Jun 23, 2023

Uh oh!

hrstoyanov commented Jun 28, 2023

Uh oh!

lvca commented Jun 28, 2023

Uh oh!

lvca commented Jun 29, 2023 •

edited

Loading

Uh oh!

lvca commented Jun 30, 2023

Uh oh!

codacy-production bot commented Jul 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lvca commented Jun 23, 2023

Uh oh!

lvca commented Jun 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lvca commented Jun 23, 2023

Uh oh!

hrstoyanov commented Jun 28, 2023

Uh oh!

lvca commented Jun 28, 2023

Uh oh!

lvca commented Jun 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lvca commented Jun 30, 2023

Uh oh!

codacy-production bot commented Jul 3, 2023

Coverage summary from Codacy

See your quality gate settings Change summary preferences

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lvca commented Jun 23, 2023 •

edited

Loading

lvca commented Jun 29, 2023 •

edited

Loading