Add customizable metadata field to MemoryRecord and memory retrieval with embeddings is optional#425
Merged
adrianwyatt merged 39 commits intomicrosoft:mainfrom Apr 13, 2023
Conversation
…rning vector optional, and in progres updating QdrantMemoryStore and unit tests
… disposal of handles and set the store to disable pooling; unit test are passing
…ary to use. Verified all samples are functional
…rning vector optional, and in progres updating QdrantMemoryStore and unit tests
shawncal
previously approved these changes
Apr 12, 2023
…rning vector optional, and in progres updating QdrantMemoryStore and unit tests
… disposal of handles and set the store to disable pooling; unit test are passing
72d7d77 to
8551328
Compare
shawncal
reviewed
Apr 12, 2023
dluc
reviewed
Apr 12, 2023
samples/dotnet/kernel-syntax-examples/Example23_ReadOnlyMemoryStore.cs
Outdated
Show resolved
Hide resolved
5 tasks
adrianwyatt
requested changes
Apr 13, 2023
dotnet/src/Connectors/Connectors.Memory.CosmosDB/CosmosMemoryStore.cs
Outdated
Show resolved
Hide resolved
dotnet/src/SemanticKernel/Planning/SKContextPlanningExtensions.cs
Outdated
Show resolved
Hide resolved
dotnet/src/SemanticKernel/Planning/SKContextPlanningExtensions.cs
Outdated
Show resolved
Hide resolved
samples/dotnet/kernel-syntax-examples/Example23_ReadOnlyMemoryStore.cs
Outdated
Show resolved
Hide resolved
adrianwyatt
approved these changes
Apr 13, 2023
dehoward
pushed a commit
to lemillermicrosoft/semantic-kernel
that referenced
this pull request
Jun 1, 2023
…with embeddings is optional (microsoft#425) ### Motivation and Context This PR addresses 3 issues. 1. MemoryRecord now has a ValueString field - this is to support customizable metadata from the user. It is up to the user to serialize and deserialize this information. Closes microsoft#202 2. Retrieving the embedding from an IMemoryStore is optional and defaults to false. In the vast majority of cases, it is unnecessary and wasteful to pass the vector around the application, especially over a network. It is primarily used for similarity comparison which performed server-side for vector optimized data stores (ex: Qdrant). Closes microsoft#202 3. Refactored SqliteMemoryStore so that connections are disposed better. There were multiple cases of residual connections causing intermittent, unexpected unit test behavior. Closes microsoft#379, Closes microsoft#430 ### Description - `IMemoryStore` `Get` and `GetNearest` calls have a new boolean parameter called `withEmbedding`. This indicates whether or not the response should include the vector associated with the memory. For all existing memory connectors, this defaults to false. This boolean is propagated to `ISemanticTextMemory` `Get` and `Search` calls as well. - `MemoryRecord` has an optional field called `valueString` to allow the user to supply a custom metadata string. This is field is added to `ISemanticTextMemory` record creation as well. - Connectors.Sqlite.Memory `Database` is no longer a static class. The connection is supplied by the `SqliteMemoryStore` which also ensures its proper disposal. - Fixed a string interpolation bug in `SqliteMemoryStore` called out in multiple issues referenced above. - Added a few missing XML docs where necessary. - Added tests for `valueString` and `withEmbeddings`. - Some files not included in this PR got picked up with `dotnet format` and `codecleanup`
golden-aries
pushed a commit
to golden-aries/semantic-kernel
that referenced
this pull request
Oct 10, 2023
…with embeddings is optional (microsoft#425) ### Motivation and Context This PR addresses 3 issues. 1. MemoryRecord now has a ValueString field - this is to support customizable metadata from the user. It is up to the user to serialize and deserialize this information. Closes microsoft#202 2. Retrieving the embedding from an IMemoryStore is optional and defaults to false. In the vast majority of cases, it is unnecessary and wasteful to pass the vector around the application, especially over a network. It is primarily used for similarity comparison which performed server-side for vector optimized data stores (ex: Qdrant). Closes microsoft#202 3. Refactored SqliteMemoryStore so that connections are disposed better. There were multiple cases of residual connections causing intermittent, unexpected unit test behavior. Closes microsoft#379, Closes microsoft#430 ### Description - `IMemoryStore` `Get` and `GetNearest` calls have a new boolean parameter called `withEmbedding`. This indicates whether or not the response should include the vector associated with the memory. For all existing memory connectors, this defaults to false. This boolean is propagated to `ISemanticTextMemory` `Get` and `Search` calls as well. - `MemoryRecord` has an optional field called `valueString` to allow the user to supply a custom metadata string. This is field is added to `ISemanticTextMemory` record creation as well. - Connectors.Sqlite.Memory `Database` is no longer a static class. The connection is supplied by the `SqliteMemoryStore` which also ensures its proper disposal. - Fixed a string interpolation bug in `SqliteMemoryStore` called out in multiple issues referenced above. - Added a few missing XML docs where necessary. - Added tests for `valueString` and `withEmbeddings`. - Some files not included in this PR got picked up with `dotnet format` and `codecleanup`
golden-aries
pushed a commit
to golden-aries/semantic-kernel
that referenced
this pull request
Oct 24, 2023
### Motivation and Context Fix for issue microsoft#425 The older API version was no longer working. It allows for deploying a new Azure OpenAI resource. ### Description Updates the API version and SKU of the Azure OpenAI ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [Contribution Guidelines](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Gil LaHaye <gillahaye@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This PR addresses 3 issues.
ConnectAsync(Connectors.Memory.Sqlite/Database.cs) #379, Closes Bad SQLite database name Assignees: kbeaugrand Labels: #430Description
IMemoryStoreGetandGetNearestcalls have a new boolean parameter calledwithEmbedding. This indicates whether or not the response should include the vector associated with the memory. For all existing memory connectors, this defaults to false. This boolean is propagated toISemanticTextMemoryGetandSearchcalls as well.MemoryRecordhas an optional field calledvalueStringto allow the user to supply a custom metadata string. This is field is added toISemanticTextMemoryrecord creation as well.Databaseis no longer a static class. The connection is supplied by theSqliteMemoryStorewhich also ensures its proper disposal.SqliteMemoryStorecalled out in multiple issues referenced above.valueStringandwithEmbeddings.dotnet formatandcodecleanupContribution Checklist
dotnet format