Skip to content

Conversation

@sjpotter
Copy link
Contributor

@sjpotter sjpotter commented Mar 9, 2023

Previously we would run the module command filters even upon blocked command reprocessing. This could modify the command, and it's args. This is irrelevant in the context of a command being reprocessed (it already went through the filters), as well as breaks the crashed command lookup that exists in the case of a reprocessed command.

fixes #11894.

Previously we would run the module command filters even upon blocked command reprocessing.  This could modify the command, and it's args.  This is irrelevant in the context of a command being reprocessed (it already went through the filters), as well as breaks the cahced command lookup that exists in the case of a reprocessed command.

fixes redis#11894.
@ranshid
Copy link
Contributor

ranshid commented Mar 9, 2023

@sjpotter can we add a test for that specific case as you mentioned here

@sjpotter
Copy link
Contributor Author

sjpotter commented Mar 9, 2023

trying to work on it right now, trying to figure out, how to get it to work in tcl, my command filter (added to commandfilter.c) is

/* Filter to protect against Bug #11894 reappearing */
void CommandFilter_BlpopFilter(RedisModuleCommandFilterCtx *filter)
{
    if (RedisModule_CommandFilterArgsCount(filter) != 3)
        return;

    const RedisModuleString *arg = RedisModule_CommandFilterArgGet(filter, 0);
    size_t arg_len;
    const char *arg_str = RedisModule_StringPtrLen(arg, &arg_len);

    if (arg_len != 5 || memcmp(arg_str, "blpop", 5))
        return;

    blpop_count++;

    if (blpop_count % 2 != 0)
        return;

    RedisModule_CommandFilterArgInsert(filter, 0, RedisModule_CreateString(NULL, "--inserted-before--", 19));
}

@sjpotter
Copy link
Contributor Author

sjpotter commented Mar 9, 2023

actually, blpop isn't a good case, as it always gets timeout from last arg. need to use brpoplpush as it gets it from a fixed position arg.

@ranshid
Copy link
Contributor

ranshid commented Mar 9, 2023

actually, blpop isn't a good case, as it always gets timeout from last arg. need to use brpoplpush as it gets it from a fixed position arg.

you can also use BLMPOP... easier still

@sjpotter
Copy link
Contributor Author

sjpotter commented Mar 9, 2023

added a test, but while my new test passes, older existing tests are seeing failures, unsure why at the moment.


brpoplpush_count++;

if (brpoplpush_count % 2 != 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this methodology. This is for tests purpose so there is no real harm, but in case the same function will be used in cases of timeout brpoplpush the counter will be messed up. can you just do an RM_CALL to incr some variable and check it's value after the client is unblocked?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I agree that its not great for general purpose (heck, 2 blockers, the 2nd wouldn't necc execute to perhaps the 3rd pass through). The thought was just making some easiy predictable for testing purposes. anything else just seems to complicate it beyond what's needed for testing purposes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can make one that reverses the direction.
i.e. changes BLMOVE RIGHT LEFT to BLMOVE LEFT RIGHT (swap the two arguments).
then to wake up the command we push two elements to the list, and watch which one is moved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oranagra , I'll try that.

@yossigo
Copy link
Collaborator

yossigo commented Mar 12, 2023

@sjpotter I wonder if we can conclude that it's always wrong to call the filter on reprocessing, as it depends on what the filter tries to achieve. Some ideas:

  • We can add a REDISMODULE_CMDFILTER_ON_UNBLOCK to indicate the callback should also be called on unblock.
  • We can also extend the filter context with a flag of whether or not we're reprocessing and add an API function to expose this.

@sjpotter
Copy link
Contributor Author

@yossigo, don't necessarily disagree with your post, some other things that could be added.

  1. extend filter context to indicate where the cmd is coming from. i.e. is it coming from a module (which one)? or from normal command processing? would deprecate the need for NOSELF, and simplify redis logic as the callback can implement it itself.

  2. as you said, is it do to being reprocessed?

  3. has it been filtered already (i.e. perhaps get both original state.

With all that said, what is the logic for running it again on reprocess? i.e. on first time through, the cmd did or didn't get modified, "reprocess" is an internal redis implementation detail, why couldn't it be implemented differently. Lets say blocking instead was handled at the blpopCommand level, it wouldn't ever reprocess. I'm trying to come up with a use case where the user would want to do that.

With that said, I'm not against the idea of being overly more flexible (and as noted, give ideas of more than can be done).

@ranshid
Copy link
Contributor

ranshid commented Mar 12, 2023

I agree with @sjpotter that the reprocessing is an internal step. this is more compatible with the way it was handled previously (before #11012)

@oranagra
Copy link
Member

i personally agree with the above. while we can do all these things, we don't have a need for that yet, and we can keep it working the same as it was before we change the internals.
@yossigo please ack.

@oranagra
Copy link
Member

I've discussed this with Yossi, we agree that the extensions described above can / should be postponed to some future date, and for now we just need to make sure the internals change doesn't break previous behavior.
i'll review the test here soon and we can merge it.


brpoplpush_count++;

if (brpoplpush_count % 2 != 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can make one that reverses the direction.
i.e. changes BLMOVE RIGHT LEFT to BLMOVE LEFT RIGHT (swap the two arguments).
then to wake up the command we push two elements to the list, and watch which one is moved.

Comment on lines +117 to +120
RedisModuleString *dir1 = RedisModule_HoldString(NULL, RedisModule_CommandFilterArgGet(filter, 3));
RedisModuleString *dir2 = RedisModule_HoldString(NULL, RedisModule_CommandFilterArgGet(filter, 4));
RedisModule_CommandFilterArgReplace(filter, 3, dir2);
RedisModule_CommandFilterArgReplace(filter, 4, dir1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to explain what you're doing here and why, not just the concern about freeing

@oranagra
Copy link
Member

@sjpotter i saw some missing comments in your command filter PR and decided to edit them instead of bothering you.
but then i realized maybe i'm missing something, so i added some comments, please clean it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[BUG] reprocessed block commands go through module command filter again, but used cached lookup

4 participants