Skip to content

Implement WITHATTRIBS for VSIM.#14065

Merged
sundb merged 4 commits into
redis:unstablefrom
antirez:vset-withattribs
May 27, 2025
Merged

Implement WITHATTRIBS for VSIM.#14065
sundb merged 4 commits into
redis:unstablefrom
antirez:vset-withattribs

Conversation

@antirez

@antirez antirez commented May 21, 2025

Copy link
Copy Markdown
Contributor

Hi, as described, this implements WITHATTRIBS, a feature requested by a few users, and indeed needed.
This was requested the first time by @rowantrollope but I was not sure how to make it work with RESP2 and RESP3 in a clean way, hopefully that's it.

The patch includes tests and documentation updates.

@snyk-io

snyk-io Bot commented May 21, 2025

Copy link
Copy Markdown

🎉 Snyk checks have passed. No issues have been found so far.

security/snyk check is complete. No issues have been found. (View Details)

license/snyk check is complete. No issues have been found. (View Details)

@sundb

sundb commented May 21, 2025

Copy link
Copy Markdown
Collaborator

hi @antirez, also need to add the new argument into modules/vector-sets/commands.

@antirez

antirez commented May 21, 2025

Copy link
Copy Markdown
Contributor Author

Thank you @sundb! Will do :)

@sundb sundb added this to Redis 8.2 May 22, 2025
@sundb sundb added release-notes indication that this issue needs to be mentioned in the release notes state:needs-doc-pr requires a PR to redis-doc repository labels May 22, 2025

@alonre24 alonre24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments

Comment thread modules/vector-sets/tests/with.py Outdated
results = self.redis.execute_command(*cmd_args)

# Results format differs depending on RESP version
resp3 = True if hasattr(self.redis, 'connection_pool') and hasattr(self.redis.connection_pool, 'response_callbacks') else False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that the proper way to test if a client is using resp3? Should we use HELLO command instead?

try:
    hello_info = r.execute_command('HELLO')
    # Check the 'proto' field in the response
    if isinstance(hello_info, dict):  # RESP3 returns a dict
        print(f"Current protocol version: {hello_info.get('proto')}")
        print("Connection is using RESP3")
    else:  # RESP2 returns an array
        # Find the 'proto' value in the array
        for i in range(0, len(hello_info), 2):
            if hello_info[i] == b'proto':
                print(f"Current protocol version: {hello_info[i+1]}")
                print("Connection is using RESP2")
                break
except Exception as e:
    print(f"Error checking protocol: {e}")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, what you suggested is better. But perhaps the Python Redis object has an explicit resp3 field actually, I need to check if it is exported.

@antirez antirez May 23, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better: I believe the test should create both RESP2 and RESP3 connections at startup, so that the tests can decide what to use and test the behavior in both sides. No test will be needed, we just use what we want based on the protocol to test, in the case when (like this one) there are divergences in the behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test updated creating a RESP3 connection and testing the behavior separately for RESP2 / RESP3.

@@ -0,0 +1,153 @@
from test import TestCase, generate_random_vector

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these tests be part of CI? Or only here to show the expected behavior?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole Python test of Vector Sets should be part of the CI, but its integration is pending right now. Somebody will do it in the next future apparently.

* item, score, attribute. For RESP3 instead item -> [score, attribute]
*/
if (resp3 && withscores && withattribs)
RedisModule_ReplyWithArray(ctx,2);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it supposed to be a map rather than an array, based on the comment above? Also, why do it only for resp3 and not for resp2 as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For RESP3, we create a map that is: map -> attribute_or_score if just one was selected, or map -> [score,attrib] if multiple will be selected. Basically the behavior will be that if just one extra information other than the element is requested, we return a simple map, otherwise we return a map mapping to an array. This is backward compatible, produces less protocol and so forth, but changes the reply format in case multiple additional items are requested. It is a tradeoff. The alternative would be that the case case would return item -> [score], a single item array, which is a bit odd.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks

Comment thread modules/vector-sets/vset.c Outdated
hnsw_release_read_slot(vset->hnsw,slot);

if (withscores)
if (resp3 && (withscores || withattribs)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why testing if resp3 here? The outer map was opened either way, no?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because for RESP3, we always return a map, and for RESP3 we always return a flat array of item, score, attrib, ..., so the two code paths have different logic. In Redis you are not used to see this explicit check because we don't have any other data type that has multiple attributes attached to the same set-alike element. For instance with sorted sets we just have the score.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. In that case, the response should start with the same logic, right? Currently we have

    /* Return results */
    if (withscores)
        RedisModule_ReplyWithMap(ctx, REDISMODULE_POSTPONED_LEN);
    else
        RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_LEN);

but I believe it should be:

    /* Return results */
    if (resp3 && (withscores || withattribs))
        RedisModule_ReplyWithMap(ctx, REDISMODULE_POSTPONED_LEN);
    else
        RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_LEN);

@antirez antirez May 27, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alonre24 I think you are right, thanks, probably it was impossible to catch this with tests, since I believe that the functions were refactored at some point into moduleReplyWithCollection(), so now basically if the reply is postponed, calling one or the other is exactly the same, and later the calls that set the length are the correct ones, so it works regardless. Verifying and pushing the fix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alonre24 fix pushed. Indeed right now RedisModule_ReplyWithMap() or RedisModule_ReplyWithArray() are equivalent if the reply is postponed, so everything worked but was conceptually wrong anyway. Fixed and refactored to make it more explicit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, it makes sense.
I Approved the PR

@sundb sundb merged commit 0ac822e into redis:unstable May 27, 2025
16 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in Redis 8.2 May 27, 2025
Comment on lines +295 to 300
{
"token": "WITHATTRIBS",
"name": "withattribs",
"type": "pure-token",
"optional": true
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @antirez does it belong to VSIM not VLINKS?

YaacovHazan pushed a commit to YaacovHazan/redis that referenced this pull request Jun 30, 2025
Hi, as described, this implements WITHATTRIBS, a feature requested by a
few users, and indeed needed.
This was requested the first time by @rowantrollope but I was not sure
how to make it work with RESP2 and RESP3 in a clean way, hopefully
that's it.

The patch includes tests and documentation updates.
@YaacovHazan YaacovHazan mentioned this pull request Jul 6, 2025
YaacovHazan pushed a commit that referenced this pull request Jul 6, 2025
Hi, as described, this implements WITHATTRIBS, a feature requested by a
few users, and indeed needed.
This was requested the first time by @rowantrollope but I was not sure
how to make it work with RESP2 and RESP3 in a clean way, hopefully
that's it.

The patch includes tests and documentation updates.
@elena-kolevska

Copy link
Copy Markdown

@redis/client-developers fyi. We need to add this to the client libraries.

sundb pushed a commit that referenced this pull request Jul 16, 2025
# Description
The `WITHATTRIBS` token was incorrectly documented under the `vlinks`
command in #14065
@YaacovHazan YaacovHazan moved this from Todo to Done in Redis 8.0 Backport Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes indication that this issue needs to be mentioned in the release notes state:needs-doc-pr requires a PR to redis-doc repository

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants