.Net MEVD: LINQ-based filtering by roji · Pull Request #10273 · microsoft/semantic-kernel

roji · 2025-01-23T13:44:08Z

LINQ-based filtering

This implements LINQ-based filtering for all current implementations of IVectorStore (with the exception of Pinecone, blocking on #10415). This implements support for:

Equality and inequality
Comparison (GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual)
Logical operators (and, or, not)
Contains over collection field (r => r.Tags.Contains("foo"))
Contains over inline collection (r => new[] { "foo", "bar" }.Contains(r.String))

Notably missing at the moment is support for VectorStoreGenericDataModel in LINQ querying; I've opened #10468 and will do this a bit later, with additional planned work on the generic data model.

API naming

Vector databases consistency refer to this feature as filter/filtering; in some cases (e.g. Chroma), where is used, following SQL nomenclature:

Qdrant
Azure AI Search
Weaviate
Milvus
Pinecone
Chroma (the term where is used in the API, but filtering is also used in the docs)

It seems the choice of exposing this via a VectorSearchOptions.Filter is indeed the correct one. Since we want to maintain backwards compatibility for existing users and preserve the current legacy filtering mechanism, this PR must introduce a separate, temporary NewFilter for the new LINQ-based filteirng. When releasing the stable version, we'll need to remove the old Filter property rename NewFilter to Filter (breaking users which switched to NewFilter).

Custom mapping

The samples currently contain a MappingVectorStoreRecordCollection, which is a decorator transforming a public, user-facing record type to another internal one - the user provides two delegates to handle the conversion from the public record type to the internal, and vice versa. Such conversions become considerably more complex, since the user provides an expression tree referencing public record properties.

To make this work, the user would need to provide some logic to translate property accesses on the public record type to the internal one. For flat records, this could be a simple API where the user simply implements some interface translating public to internal; but for hierarchical records this gets a bit more complicated. I'd suggest deferring such advanced, arbitrary mapping scenarios to post-release. In the meantime, I've commented out MappingVectorStoreRecordCollection (and also Step4_NonStringKey_VectorStore) from the samples.

Note that the problem here doesn't come from LINQ per se, but from the fact that the user specifies model property names (the legacy search API had them specify storage property names instead, side-stepping the problem but causing an inconsistent API where sometimes model properties are used, and sometimes storage properties).

Integration test suite

In addition, this implements new integration test projects for all IVectorStore connectors, implementing most of #10194 (comment):

A shared FiltersTestBase class defines all LINQ filter tests, and is extended and used by all providers (link). This can serve as the basic "spec" for what filters are supported across connector (subject to database support, of course).
Each connector has its own separate project
Testcontainers are used for Mongo, PostgreSQL, Qdrant, Redis, Sqlite, and Weaviate. Azure AI Search, CosmosNoSQL and CosmosMongoDB require configuration (environment variables, settings.json) to provide endpoint information, and tests are skipped if that's not provided.

/cc @westey-m @dmytrostruk

Closes #10156
Does most of #10194

dotnet/src/Connectors/Connectors.Memory.Qdrant/QdrantFilterTranslator.cs

roji · 2025-01-23T13:51:14Z

dotnet/src/Connectors/VectorData.Abstractions/VectorSearch/IVectorizableTextSearch.cs

    Task<VectorSearchResults<TRecord>> VectorizableTextSearchAsync(
        string searchText,
-        VectorSearchOptions? options = default,
+        VectorSearchOptions<TRecord>? options = default,


Making VectorSearchOptions generic over TRecord isn't trivial... Modern C# users can use target-typed new, and in that case this doesn't interfere in the method calling. But people not doing target-typed new would have to explicitly specify the full type every time (even if they don't use filtering at all), which maybe isn't ideal. The alternative would be to move the LINQ filter to an optional parameter directly on the method (no strong feelings from my side here).

Since we are thinking of also making property selection an expression, that may result in a few more optional parameters on the method signature, to avoid making options generic. For hybrid search it would be 3 additional, which isn't great, especially considering that 2 of those would very rarely be used, i.e. only if there is more than one dense vector or more than one fulltextsearch property on the record.

dotnet/SK-dotnet.sln

dotnet/src/Connectors/VectorData.Abstractions/VectorSearch/VectorSearchOptions.cs

dotnet/src/VectorDataTests/VectorDataSpecificationTests/Filter/BasicFilterTestsBase.cs

dotnet/src/VectorDataTests/QdrantTests/Support/TestContainer/QdrantBuilder.cs

dotnet/src/VectorDataTests/QdrantTests/Support/TestContainer/QdrantConfiguration.cs

dotnet/src/VectorDataTests/QdrantTests/Support/TestContainer/QdrantContainer.cs

westey-m · 2025-01-23T16:44:09Z

@roji, a good scenario to also include in early exploration would be tag based operations, e.g. where there is an array of strings (tags) in a field in the database, and we want to check if one or more out of tags we provide match those.
We have received consistent feedback that this is a popular scenario for vector search filtering.

dotnet/src/VectorDataTests/VectorDataSpecificationTests/Filter/BasicFilterTestsBase.cs

roji · 2025-01-23T22:30:51Z

a good scenario to also include in early exploration would be tag based operations, e.g. where there is an array of strings (tags) in a field in the database, and we want to check if one or more out of tags we provide match those.
We have received consistent feedback that this is a popular scenario for vector search filtering.

@westey-m thanks - will do that, it should be easily representible in LINQ by doing b => b.Tags.Contains("foo"). Maybe let's talk tomorrow about how that's represented e.g. in Qdrant (just because I'm currently focusing on it).

adamsitnik

LGTM, great work @roji ! I am very happy to see the Testcontainers in action. Also, they way you have implemented the base class for tests is very clever!

...c/VectorDataIntegrationTests/CosmosMongoDBIntegrationTests/Support/CosmosMongoDBTestStore.cs

adamsitnik · 2025-02-11T14:01:39Z

dotnet/src/VectorDataIntegrationTests/Directory.Build.props

@@ -0,0 +1,15 @@
+<Project>
+  <Import Project="../../Directory.Build.props" />


It's great that we have a props file, so we can move all the things that would be common for all test projects here 👍

Yep. There's some opportunity to refactor common stuff, but I didn't want to go too far in this already-huge PR.

adamsitnik · 2025-02-11T14:03:20Z

dotnet/src/VectorDataIntegrationTests/InMemoryIntegrationTests/InMemoryIntegrationTests.csproj

+  <ItemGroup>
+    <PackageReference Include="xunit"/>
+    <PackageReference Include="xunit.runner.visualstudio">
+      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
+      <PrivateAssets>all</PrivateAssets>
+    </PackageReference>
+    <PackageReference Include="Microsoft.NET.Test.Sdk"/>
+  </ItemGroup>


In one of the future refactorings, we could move this part to the Directory.Build.props file

dotnet/src/VectorDataIntegrationTests/MongoDBIntegrationTests/Support/MongoDBTestStore.cs

dotnet/src/VectorDataIntegrationTests/VectorDataIntegrationTests/Filter/BasicFilterTestsBase.cs

dotnet/src/VectorDataIntegrationTests/VectorDataIntegrationTests/Filter/FilterFixtureBase.cs

adamsitnik · 2025-02-11T14:36:58Z

.../src/VectorDataIntegrationTests/VectorDataIntegrationTests/Xunit/ConditionalFactAttribute.cs

+
+[AttributeUsage(AttributeTargets.Method)]
+[XunitTestCaseDiscoverer("VectorDataSpecificationTests.Xunit.ConditionalFactDiscoverer", "VectorDataIntegrationTests")]
+public sealed class ConditionalFactAttribute : FactAttribute;


Have you considered importing the package that dotnet repo use to get ConditionalFact and other xUnit extensions?

I remember there's something at least for internal .NET consumption, but I wasn't aware that there was a full public package meant for public consumption by anyone - if you point me to it we can consider using that (FWIW even EF has a copy of these internally). Though at the end of the day it's just a few classes which will basically never change, it's probably not super important.

dmytrostruk · 2025-02-11T15:43:23Z

@roji There is small formatting warning in WeaviateTestStore.cs file:

If I remove it though, it says it can't find HttpClient for net472:

For simplicity of this PR, I think we can remove net472 from target frameworks and handle it in scope of separate task. We also need net472 for other projects and not only for vector stores.

roji · 2025-02-11T15:55:22Z

For simplicity of this PR, I think we can remove net472 from target frameworks and handle it in scope of separate task. We also need net472 for other projects and not only for vector stores.

Yeah... Though I already did most of the work around this - I'll get rid of the last formatting stuff and merge, at the point where I am...

.../src/VectorDataIntegrationTests/VectorDataIntegrationTests/VectorDataIntegrationTests.csproj

dmytrostruk

Small improvements, but in general looks awesome!

dotnet/Directory.Build.props

dotnet/SK-dotnet.sln

...ors.AzureCosmosDBNoSQL.UnitTests/AzureCosmosDBNoSQLVectorStoreCollectionQueryBuilderTests.cs

westey-m · 2025-02-11T17:40:38Z

dotnet/samples/Concepts/Memory/VectorStoreLangchainInterop/QdrantFactory.cs

                return (collection as IVectorStoreRecordCollection<TKey, TRecord>)!;
            }

+#if DISABLED_FOR_NOW // TODO: See note on MappingVectorStoreRecordCollection


Should it not be #if !DISABLED_FOR_NOW?

Well this basically is just a glorified comment (I did it with #if instead of /* */ mostly because the latter doesn't nest). If I negate, we'd have to define DISABLED_FOR_NOW in the csproj for this to be commented out...

westey-m · 2025-02-11T17:41:03Z

...et/samples/Concepts/Memory/VectorStoreLangchainInterop/MappingVectorStoreRecordCollection.cs

+// TODO: The user provides an expression tree accepting a TPublicRecord, but we require an expression tree accepting a TInternalRecord.
+// TODO: This is something that the user must provide, and is quite advanced.
+
+#if DISABLED


Should it be #if !DISABLED

Same as above

dotnet/src/SemanticKernel.Core/SemanticKernel.Core.csproj

dluc · 2025-02-11T18:04:42Z

Notably missing at the moment is support for VectorStoreGenericDataModel in LINQ querying; I've opened #10468 and will do this a bit later, with additional planned work on the generic data model.

I could not understand the implications, where/when is VectorStoreGenericDataModel used? perhaps an old abstraction that could be deprecated?

dluc · 2025-02-11T18:10:32Z

I noticed a comment and discussion about this in the PR: if I understand correctly LINQ filters are expressed using internal storage field names, could you confirm?

If needed, is there a possibility for mapping external names to internal names? For example, let's assume I build a solution that works with TAGS, and they are stored in different ways, e.g.

a list of strings in Postgres, field name "tags"
a list of string fields in Redis, field names "tag1" "tag2" "tag3" ...
a JSON property in MongoDb, field name "doc.props.tags"

In this scenario would I be able to transform the incoming LINQ filter on "tags" to another LINQ filter targeting my storage of choice?

dluc · 2025-02-11T18:18:11Z

dotnet/src/Connectors/Connectors.Memory.Postgres/IPostgresVectorStoreCollectionSqlBuilder.cs

    /// <param name="vectorValue">The vector to match.</param>
-    /// <param name="filter">The filter conditions for the query.</param>
+    /// <param name="legacyFilter">The filter conditions for the query.</param>
+    /// <param name="newFilter">The filter conditions for the query.</param>


Renaming parameters is a notable breaking change. Would it make sense to tackle all the breaking changes at once, including removing the old filter and updating the type, to streamline the transition?

This is something we discussed quite extensively recently... The decision was made to obsolete but preserve the current search mechanism, to prevent breakage for current users. We'd remove the legacy search API before releasing, and rename NewFilter to Filter at that point (this creates more churn for users keeping up to date and using the new API).

dluc · 2025-02-11T18:21:29Z

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreCollectionSqlBuilder.cs

+#pragma warning disable CS0618 // VectorSearchFilter is obsolete
+        var (where, parameters) = (oldFilter: legacyFilter, newFilter) switch
+        {
+            (not null, not null) => throw new ArgumentException("Either Filter or NewFilter can be specified, but not both"),


nit: use nameof(Filter) and nameof(NewFilter) in case params are renamed

Good suggestion - I'll pass in this specific case only because Filter will be removed soon (before release) anyway and this whole piece of code will go away.

dluc · 2025-02-11T18:24:27Z

dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresFilterTranslator.cs

+
+    private readonly StringBuilder _sql = new();
+
+    internal (string Clause, List<object> Parameters) Translate(


do you think there's room for SQL injection, considering that the resulting string is merged into a SQL statement?

Absolutely (and that's very important). There's a test exercising an input with a single quote to check exactly for this. I have a note to do a general comprehensive pass to make sure everything is airtight - stuff like property names isn't currently safe (those generally aren't user input, but I still need to make them airtight).

dluc · 2025-02-11T18:34:13Z

Could you confirm that NOT works also with Contains? example

NOT Contains over collection field (r => !r.Tags.Contains("foo"))

thanks!

westey-m · 2025-02-11T20:49:33Z

dotnet/src/VectorDataIntegrationTests/RedisIntegrationTests/Support/RedisTestStore.cs

+        await this._container.StartAsync();
+        var redis = await ConnectionMultiplexer.ConnectAsync($"{this._container.Hostname}:{this._container.GetMappedPublicPort(6379)},connectTimeout=60000,connectRetry=5");
+        this._database = redis.GetDatabase();
+        this._defaultVectorStore = new(this._database);


We should make sure to test both instances of redis: Json and Hashset. It's possible to choose the flavour by setting the StorageType option on RedisVectorStoreOptions. They do differ in some non-obvious ways, especially with regards to how you specify property names.

Totally missed that, thanks... I pushed tests for both. Depending on the exact differences, it might make sense to just treat these as two databases test-wise and have two projects (but for now fixtures and tests are duplicated in the same projects).

Query-wise there are some unsupported scenarios in HashSet compared to JSON (array fields, null values), but otherwise the tests pass.

roji · 2025-02-11T21:57:22Z

@dluc

Notably missing at the moment is support for VectorStoreGenericDataModel in LINQ querying; I've opened #10468 and will do this a bit later, with additional planned work on the generic data model.

I could not understand the implications, where/when is VectorStoreGenericDataModel used? perhaps an old abstraction that could be deprecated?

VectorStoreGenericDataModel is basically a way to use this abstraction without a strongly-typed .NET type (e.g. for dynamic scenarios). This is already supported in MEVD, but this PR doesn't add support for it in the new filtering mechanism. This is absolutely something I intend to do.

I noticed a comment and discussion about this in the PR: if I understand correctly LINQ filters are expressed using internal storage field names, could you confirm?

No - the LINQ filter expression references properties on the user's .NET type, and so references regular .NET properties.

For example, in collection.VectorizedSearchAsync(vector, new() { NewFilter = r => r.Age > 8 }), Age is a .NET property on a real .NET type (represented here by the lambda parameter r). MEVD allows mapping .NET properties to arbitrary storage names (the names that you actually see in the database), and that's fully compatible with the new LINQ filtering (the translator goes through an internal translator table from user-facing property names to storage properties).

If needed, is there a possibility for mapping external names to internal names? For example, let's assume I build a solution that works with TAGS, and they are stored in different ways, e.g.

a list of strings in Postgres, field name "tags"
a list of string fields in Redis, field names "tag1" "tag2" "tag3" ...
a JSON property in MongoDb, field name "doc.props.tags"
In this scenario would I be able to transform the incoming LINQ filter on "tags" to another LINQ filter targeting my storage of choice?

For all of these scenarios, your .NET type would simply have a regular List<string> property called Tags; referencing it in the LINQ filter and e.g. calling Contains on that property should work simply work on all these databases. One note though: AFAIK in Redis, this would not be mapped to multiple string fields, but rather with a single Tag field; this already fully works with the Redis connector. This scenario is exercised by this test.

Could you confirm that NOT works also with Contains? example

Yes - it does in general. Specific database support varies sometimes (e.g. MongoDB has some general issues with NOT in vector search pre-filters), but it works fine on most databases.

roji · 2025-02-11T22:44:56Z

BTW @dluc we're definitely interested in feedback about any other specific filtering features you might be interested, don't hesitate to let us know.

dluc · 2025-03-13T04:12:03Z

@roji this is a scenario I'm about to test.

I'm curious to see this in action since I don't have classes modeling data in storage, every definition is set at runtime. I guess internal field names and external field names will match in every scenario, thus it should just work™️

I noticed a comment and discussion about this in the PR: if I understand correctly LINQ filters are expressed using internal storage field names, could you confirm?

No - the LINQ filter expression references properties on the user's .NET type, and so references regular .NET properties.

roji · 2025-03-13T06:58:58Z

I'm curious to see this in action since I don't have classes modeling data in storage, every definition is set at runtime. I guess internal field names and external field names will match in every scenario, thus it should just work

@dluc if I understand correctly, does that mean you using VectorStoreGenericDataModel? If so, note that the LINQ-based filteirng hasn't yet been implemented for that - #10468 tracks that. Full support for dynamic scenarios is absolutely high our list, but some fixes probably need to happen first (see #10802).

So long story short, for dynamic scenarios you'll likely need to wait a bit longer until everything is fully in place.

markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Jan 23, 2025

github-actions bot changed the title ~~WIP on LINQ-based criteria filtering~~ .Net: WIP on LINQ-based criteria filtering Jan 23, 2025

roji had a problem deploying to integration January 23, 2025 13:44 — with GitHub Actions Failure

roji commented Jan 23, 2025

View reviewed changes

dmytrostruk reviewed Jan 23, 2025

View reviewed changes

roji had a problem deploying to integration January 23, 2025 15:40 — with GitHub Actions Failure

roji had a problem deploying to integration January 23, 2025 16:25 — with GitHub Actions Failure

westey-m reviewed Jan 23, 2025

View reviewed changes

dotnet/src/VectorDataTests/VectorDataSpecificationTests/Filter/BasicFilterTestsBase.cs Outdated Show resolved Hide resolved

roji changed the base branch from main to feature-vector-data-preb1 January 24, 2025 18:18

roji had a problem deploying to integration January 24, 2025 18:29 — with GitHub Actions Failure

roji had a problem deploying to integration January 24, 2025 19:32 — with GitHub Actions Failure

roji had a problem deploying to integration January 25, 2025 00:29 — with GitHub Actions Failure

roji force-pushed the LinqFiltering branch from ef07816 to 20a2aea Compare January 25, 2025 00:39

roji had a problem deploying to integration January 25, 2025 00:39 — with GitHub Actions Failure

roji had a problem deploying to integration January 25, 2025 07:58 — with GitHub Actions Failure

roji had a problem deploying to integration January 25, 2025 11:17 — with GitHub Actions Failure

roji had a problem deploying to integration January 26, 2025 11:24 — with GitHub Actions Failure

roji had a problem deploying to integration January 26, 2025 12:50 — with GitHub Actions Failure

roji had a problem deploying to integration January 26, 2025 13:37 — with GitHub Actions Failure

roji had a problem deploying to integration January 29, 2025 21:56 — with GitHub Actions Failure

roji had a problem deploying to integration January 30, 2025 20:00 — with GitHub Actions Failure

roji had a problem deploying to integration January 31, 2025 18:58 — with GitHub Actions Failure

roji had a problem deploying to integration February 1, 2025 22:15 — with GitHub Actions Failure

roji had a problem deploying to integration February 4, 2025 22:56 — with GitHub Actions Failure

roji mentioned this pull request Feb 6, 2025

[MEVD] Specification/conformance integration test suite for providers #10194

Closed

roji had a problem deploying to integration February 6, 2025 17:49 — with GitHub Actions Error

adamsitnik approved these changes Feb 11, 2025

View reviewed changes

Final formatting fixes

161ee58

dmytrostruk reviewed Feb 11, 2025

View reviewed changes

.../src/VectorDataIntegrationTests/VectorDataIntegrationTests/VectorDataIntegrationTests.csproj Outdated Show resolved Hide resolved

Make non-package again

eb07d55

dmytrostruk approved these changes Feb 11, 2025

View reviewed changes

dotnet/Directory.Build.props Outdated Show resolved Hide resolved

dotnet/SK-dotnet.sln Outdated Show resolved Hide resolved

...ors.AzureCosmosDBNoSQL.UnitTests/AzureCosmosDBNoSQLVectorStoreCollectionQueryBuilderTests.cs Outdated Show resolved Hide resolved

westey-m reviewed Feb 11, 2025

View reviewed changes

dotnet/src/SemanticKernel.Core/SemanticKernel.Core.csproj Outdated Show resolved Hide resolved

dluc reviewed Feb 11, 2025

View reviewed changes

dluc mentioned this pull request Feb 11, 2025

Added very first spike for NOT filtering microsoft/kernel-memory#689

Closed

westey-m reviewed Feb 11, 2025

View reviewed changes

westey-m approved these changes Feb 11, 2025

View reviewed changes

Address review comments, implement Redis HashSet tests

de13cff

This was referenced Feb 15, 2025

Python: Support BetweenFilterClause or RangeFilterClause for Redis #10391

Closed

.Net: Refactor vector store integration tests #10556

Merged

roji mentioned this pull request Mar 4, 2025

.Net: Rename VectorSearchOptions.NewFilter to Filter #10790

Merged

This was referenced Sep 20, 2025

.Net MEVD: Modify ITextSearch to react to the new LINQ-based vector search filtering #10456

Open

.Net: feat: Implement type-safe LINQ filtering for ITextSearch interface (microsoft#10456) #13175

Merged

alzarei mentioned this pull request Oct 17, 2025

.NET: Add ITextSearch<TRecord> with LINQ filtering and deprecate legacy ITextSearch (#10456) #13179

Merged

		@@ -0,0 +1,15 @@
		<Project>
		<Import Project="../../Directory.Build.props" />


		private readonly StringBuilder _sql = new();

		internal (string Clause, List<object> Parameters) Translate(

Conversation

roji commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LINQ-based filtering

API naming

Custom mapping

Integration test suite

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

westey-m commented Jan 23, 2025

Uh oh!

Uh oh!

roji commented Jan 23, 2025

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmytrostruk commented Feb 11, 2025

Uh oh!

roji commented Feb 11, 2025

Uh oh!

Uh oh!

dmytrostruk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dluc commented Feb 11, 2025

Uh oh!

dluc commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dluc Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roji commented Jan 23, 2025 •

edited

Loading

dluc commented Feb 11, 2025 •

edited

Loading

dluc Feb 11, 2025 •

edited

Loading

roji Feb 11, 2025 •

edited

Loading