… on keys (#11832)

Currently (starting at #11012) When a module is blocked on keys it sets the
CLIENT_PENDING_COMMAND flag.
However in case the module decides to unblock the client not via the regular flow
(eg timeout, key signal or CLIENT UNBLOCK command) it will attempt to reprocess the
module command and potentially blocked again.

This fix remove the CLIENT_PENDING_COMMAND flag in case blockedForKeys is
issued from module context.

In #11012, we changed the way command durations were computed to handle the same command being executed multiple times. This commit fixes some misses from that commit.

* Wait commands were not correctly reporting their duration if the timeout was reached.
* Multi/scripts/and modules with RM_Call were not properly resetting the duration between inner calls, leading to them reporting cumulative duration.
* When a blocked client is freed, the call and duration are always discarded.

This commit also adds an assert if the duration is not properly reset, potentially indicating that a report to call statistics was missed. The assert potentially be removed in the future, as it's mainly intended to detect misses in tests.

For the XREADGROUP BLOCK > scenario, there is an endless loop.
Due to redis#11012, it keep going, reprocess command -> blockForKeys -> reprocess command

The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
is problematic since the code keeps accessing the global list.

The fix is proposed by oranagra.

Fixes redis#12290

Co-authored-by: Oran Agra <oran@redislabs.com>

For the XREADGROUP BLOCK > scenario, there is an endless loop.
Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command

The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
is problematic since the code keeps accessing the global list.

Co-authored-by: Oran Agra <oran@redislabs.com>

*TL;DR*
---------------------------------------
Following the discussion over the issue [redis#7551](redis#7551)
We decided to refactor the client blocking code to eliminate some of the code duplications
and to rebuild the infrastructure better for future key blocking cases.


*In this PR*
---------------------------------------
1. reprocess the command once a client becomes unblocked on key (instead of running
   custom code for the unblocked path that's different than the one that would have run if
   blocking wasn't needed)
2. eliminate some (now) irrelevant code for handling unblocking lists/zsets/streams etc...
3. modify some tests to intercept the error in cases of error on reprocess after unblock (see
   details in the notes section below)
4. replace '$' on the client argv with current stream id. Since once we reprocess the stream
   XREAD we need to read from the last msg and not wait for new msg  in order to prevent
   endless block loop. 
5. Added statistics to the info "Clients" section to report the:
   * `total_blocking_keys` - number of blocking keys
   * `total_blocking_keys_on_nokey` - number of blocking keys which have at least 1 client
      which would like
   to be unblocked on when the key is deleted.
6. Avoid expiring unblocked key during unblock. Previously we used to lookup the unblocked key
   which might have been expired during the lookup. Now we lookup the key using NOTOUCH and
   NOEXPIRE to avoid deleting it at this point, so propagating commands in blocked.c is no longer needed.
7. deprecated command flags. We decided to remove the CMD_CALL_STATS and CMD_CALL_SLOWLOG
   and make an explicit verification in the call() function in order to decide if stats update should take place.
   This should simplify the logic and also mitigate existing issues: for example module calls which are
   triggered as part of AOF loading might still report stats even though they are called during AOF loading.

*Behavior changes*
---------------------------------------------------

1. As this implementation prevents writing dedicated code handling unblocked streams/lists/zsets,
since we now re-process the command once the client is unblocked some errors will be reported differently.
The old implementation used to issue
``UNBLOCKED the stream key no longer exists``
in the following cases:
   - The stream key has been deleted (ie. calling DEL)
   - The stream and group existed but the key type was changed by overriding it (ie. with set command)
   - The key not longer exists after we swapdb with a db which does not contains this key
   - After swapdb when the new db has this key but with different type.
   
In the new implementation the reported errors will be the same as if the command was processed after effect:
**NOGROUP** - in case key no longer exists, or **WRONGTYPE** in case the key was overridden with a different type.

2. Reprocessing the command means that some checks will be reevaluated once the
client is unblocked.
For example, ACL rules might change since the command originally was executed and
will fail once the client is unblocked.
Another example is OOM condition checks which might enable the command to run and
block but fail the command reprocess once the client is unblocked.

3. One of the changes in this PR is that no command stats are being updated once the
command is blocked (all stats will be updated once the client is unblocked). This implies
that when we have many clients blocked, users will no longer be able to get that information
from the command stats. However the information can still be gathered from the client list.

**Client blocking**
---------------------------------------------------

the blocking on key will still be triggered the same way as it is done today.
in order to block the current client on list of keys, the call to
blockForKeys will still need to be made which will perform the same as it is today:

*  add the client to the list of blocked clients on each key
*  keep the key with a matching list node (position in the global blocking clients list for that key)
   in the client private blocking key dict.
*  flag the client with CLIENT_BLOCKED
*  update blocking statistics
*  register the client on the timeout table

**Key Unblock**
---------------------------------------------------

Unblocking a specific key will be triggered (same as today) by calling signalKeyAsReady.
the implementation in that part will stay the same as today - adding the key to the global readyList.
The reason to maintain the readyList (as apposed to iterating over all clients blocked on the specific key)
is in order to keep the signal operation as short as possible, since it is called during the command processing.
The main change is that instead of going through a dedicated code path that operates the blocked command
we will just call processPendingCommandsAndResetClient.

**ClientUnblock (keys)**
---------------------------------------------------

1. Unblocking clients on keys will be triggered after command is
   processed and during the beforeSleep
8. the general schema is:
9. For each key *k* in the readyList:
```            
For each client *c* which is blocked on *k*:
            in case either:
	          1. *k* exists AND the *k* type matches the current client blocking type
	  	      OR
	          2. *k* exists and *c* is blocked on module command
	    	      OR
	          3. *k* does not exists and *c* was blocked with the flag
	             unblock_on_deleted_key
                 do:
                                  1. remove the client from the list of clients blocked on this key
                                  2. remove the blocking list node from the client blocking key dict
                                  3. remove the client from the timeout list
                                  10. queue the client on the unblocked_clients list
                                  11. *NEW*: call processCommandAndResetClient(c);
```
*NOTE:* for module blocked clients we will still call the moduleUnblockClientByHandle
              which will queue the client for processing in moduleUnblockedClients list.

**Process Unblocked clients**
---------------------------------------------------

The process of all unblocked clients is done in the beforeSleep and no change is planned
in that part.

The general schema will be:
For each client *c* in server.unblocked_clients:

        * remove client from the server.unblocked_clients
        * set back the client readHandler
        * continue processing the pending command and input buffer.

*Some notes regarding the new implementation*
---------------------------------------------------

1. Although it was proposed, it is currently difficult to remove the
   read handler from the client while it is blocked.
   The reason is that a blocked client should be unblocked when it is
   disconnected, or we might consume data into void.

2. While this PR mainly keep the current blocking logic as-is, there
   might be some future additions to the infrastructure that we would
   like to have:
   - allow non-preemptive blocking of client - sometimes we can think
     that a new kind of blocking can be expected to not be preempt. for
     example lets imagine we hold some keys on disk and when a command
     needs to process them it will block until the keys are uploaded.
     in this case we will want the client to not disconnect or be
     unblocked until the process is completed (remove the client read
     handler, prevent client timeout, disable unblock via debug command etc...).
   - allow generic blocking based on command declared keys - we might
     want to add a hook before command processing to check if any of the
     declared keys require the command to block. this way it would be
     easier to add new kinds of key-based blocking mechanisms.

Co-authored-by: Oran Agra <oran@redislabs.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>

… on keys (redis#11832)

Currently (starting at redis#11012) When a module is blocked on keys it sets the
CLIENT_PENDING_COMMAND flag.
However in case the module decides to unblock the client not via the regular flow
(eg timeout, key signal or CLIENT UNBLOCK command) it will attempt to reprocess the
module command and potentially blocked again.

This fix remove the CLIENT_PENDING_COMMAND flag in case blockedForKeys is
issued from module context.

In redis#11012, we will reprocess command when client is unblocked on keys,
In the XREADGROUP BLOCK scenario, because of the re-processing command,
we will recalculate the block timeout, causing the blocking time to be
reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in blockForKeys
we will not reset the timeout.

Fixes redis#12998.

…13004)

In #11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases: 
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes #12998.

…edis#13004)

In redis#11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases: 
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes redis#12998.

…edis#13004)

In redis#11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases:
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes redis#12998.

(cherry picked from commit 492021d)

…13004)

In #11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases:
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes #12998.

(cherry picked from commit 492021d)

…edis#13004)

In redis#11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases: 
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes redis#12998.