Skip to content

Conversation

@raz-mon
Copy link
Contributor

@raz-mon raz-mon commented Jan 13, 2025

PR: Add Mechanism for Internal Commands and Connections in Redis

This PR introduces a mechanism to handle internal commands and connections in Redis. It includes enhancements for command registration, internal authentication, and observability.

Key Features

  1. Internal Command Flag:

    • Introduced a new module command registration flag: internal.
    • Commands marked with internal can only be executed by internal connections, AOF loading flows, and master-replica connections.
    • For any other connection, these commands will appear as non-existent.
  2. Support for internal authentication added to AUTH:

    • Used by depicting the special username internal connection with the right internal password, i.e.,: AUTH "internal connection" <internal_secret>.
    • No user-defined ACL username can have this name, since spaces are not aloud in the ACL parser.
    • Allows connections to authenticate as internal connections.
    • Authenticated internal connections can execute internal commands successfully.
  3. Module API for Internal Secret:

    • Added the RedisModule_GetInternalSecret() API, that exposes the internal secret that should be used as the password for the new AUTH "internal connection" <password> command.
    • This API enables the modules to authenticate against other shards as local connections.

Notes on Behavior

  • ACL validation:

    • Commands dispatched by internal connections bypass ACL validation, to give the caller full access regardless of the user with which it is connected.
  • Command Visibility:

    • Internal commands do not appear in COMMAND <subcommand> and MONITOR for non-internal connections.
    • Internal commands are logged in the slow log, latency report and commands' statistics to maintain observability.
  • RM_Call() Updates:

    • Non-internal connections:
      • Cannot execute internal commands when the command is sent with the C flag (otherwise can).
    • Internal connections bypass ACL validations (i.e., run as the unrestricted user).
  • Internal commands' success:

    • Internal commands succeed upon being sent from either an internal connection (i.e., authenticated via the new AUTH "internal connection" <internal_secret> API), an AOF loading process, or from a master via the replication link.
      Any other connections that attempt to execute an internal command fail with the unknown command error message raised.
  • CLIENT LIST flags:

    • Added the I flag, to indicate that the connection is internal.
  • Lua Scripts:

    • Prevented internal commands from being executed via Lua scripts.

@CLAassistant
Copy link

CLAassistant commented Jan 13, 2025

CLA assistant check
All committers have signed the CLA.

@raz-mon raz-mon marked this pull request as draft January 14, 2025 06:17
@raz-mon raz-mon changed the title MOD-8537 MOD-8538 MOD-8536: Add internal connection and command mechanism Add internal connection and command mechanism Jan 14, 2025
Copy link

@MeirShpilraien MeirShpilraien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
@raz-mon looks good.
Added some comments but please notice that I did not yet finished the review all the changes.

@raz-mon raz-mon marked this pull request as ready for review January 20, 2025 08:44
The PR intoruduces a new shared secret that is shared over all
the nodes on the Redis cluster. The main idea is to leverage
the cluster bus to share a secret between all the nodes
such that later the nodes will be able to authenticate
using this secret and send internal commands to each other
(see redis#13740 for more inforamtion about internal commands).

The way the shared secret is chosen is the following:
1. Each node, when start, randomly generate its own internal
   secret.
2. Each node share its internal secret over the cluster ping messages.
3. In addition, each node also shared the secrets of the other nodes it knows about as part of the gossip protocol.
4. Node choses the secret with the minimal value (lexicographical).

The converges of the secret is as good as the topology converges.

To extend the ping messages to contain the secret, we leverage the extention machanism. Nodes that runs an older Redis version will just ignore those extentions.

Secrets are also persisted to the `node.conf` file so that on start we will have the secret right away.

Specific tests were added to verify that eventually all nodes see the secrets. In addition, a verification was added to the test infra to verify the secret on `cluster_config_consistent` and to `assert_cluster_state`
Comment on lines +242 to +246
test {Internal commands are reported in the latency report} {
# The latency report should contain the internal command
set report [r latency histogram internalauth.internalcommand]
assert_match {*internalauth.internalcommand*} $report
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably do it now via an internal connection, we are still internal client from the test above this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow your meaning of doing it now via an internal connection.
I can switch to a non-internal connection so that we make sure we get this there as well

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can switch to a non-internal connection so that we make sure we get this there as well

Yes this is what I meant.

}
}

start_cluster 1 0 [list config_lines $modules] {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you do not need cluster deployment for those tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly for the regular internal connection authentication flow, otherwise can be reached via the non-standard debug command.
Any reason not to use a cluster and the regular flow?

Copy link

@MeirShpilraien MeirShpilraien Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be faster to deploy, I believe we better use this and just promote the connection with debug command. Also simplify the test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I can change to use a standalone server

Copy link

@MeirShpilraien MeirShpilraien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Added a few comments on the tests.

Copy link

@MeirShpilraien MeirShpilraien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there, 2 small comments.

Copy link

@MeirShpilraien MeirShpilraien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@oranagra oranagra added the release-notes indication that this issue needs to be mentioned in the release notes label Feb 5, 2025
@YaacovHazan YaacovHazan merged commit 04589f9 into redis:unstable Feb 5, 2025
18 checks passed
MeirShpilraien pushed a commit to RedisGears/LibMR that referenced this pull request Feb 5, 2025
The PR makes changes to use the internal command capabilities introduced on: redis/redis#13740.
The main idea is to use the new redismodule API that provides internal secret for internal shard communication.

This means that the following is no longer needed:
1. The new user configuration.
2. The internal ACL category

All the commands are marked with the `internal` flag (if we see that the internal commands feature is supported on the current Redis version we run on). In addition, the connection between the shards uses `INTERNALAUTH` with the given internal secret so it can run all the internal commands. Regular user can not run any internal command.

The PR require the following PR's to be merge:
1. redis/redis#13740
2. RedisLabsModules/redismodule-rs#407

The tests was modified to avoid giving password to the test module in case it detects that the internal commands feature exists. In such case, avoiding giving the password to the module, make sure that LibMR actually using the `INTERNALAUTH` command with the internal secret to authenticate.
@sundb sundb added this to Redis 8.0 Aug 15, 2025
@sundb sundb moved this from Todo to Done in Redis 8.0 Aug 15, 2025
funny-dog pushed a commit to funny-dog/redis that referenced this pull request Sep 17, 2025
The PR introduces a new shared secret that is shared over all the nodes
on the Redis cluster. The main idea is to leverage the cluster bus to
share a secret between all the nodes such that later the nodes will be
able to authenticate using this secret and send internal commands to
each other (see redis#13740 for more information about internal commands).

The way the shared secret is chosen is the following:
1. Each node, when start, randomly generate its own internal secret.
2. Each node share its internal secret over the cluster ping messages.
3. If a node gets a ping message with secret smaller then his current
secret, it embrace it.
4. Eventually all nodes should embrace the minimal secret

The converges of the secret is as good as the topology converges.

To extend the ping messages to contain the secret, we leverage the
extension mechanism. Nodes that runs an older Redis version will just
ignore those extensions.

Specific tests were added to verify that eventually all nodes see the
secrets. In addition, a verification was added to the test infra to
verify the secret on `cluster_config_consistent` and to
`assert_cluster_state`.
funny-dog pushed a commit to funny-dog/redis that referenced this pull request Sep 17, 2025
# PR: Add Mechanism for Internal Commands and Connections in Redis

This PR introduces a mechanism to handle **internal commands and
connections** in Redis. It includes enhancements for command
registration, internal authentication, and observability.

## Key Features

1. **Internal Command Flag**:
   - Introduced a new **module command registration flag**: `internal`.
- Commands marked with `internal` can only be executed by **internal
connections**, AOF loading flows, and master-replica connections.
- For any other connection, these commands will appear as non-existent.

2. **Support for internal authentication added to `AUTH`**:
- Used by depicting the special username `internal connection` with the
right internal password, i.e.,: `AUTH "internal connection"
<internal_secret>`.
- No user-defined ACL username can have this name, since spaces are not
aloud in the ACL parser.
   - Allows connections to authenticate as **internal connections**.
- Authenticated internal connections can execute internal commands
successfully.

4. **Module API for Internal Secret**:
- Added the `RedisModule_GetInternalSecret()` API, that exposes the
internal secret that should be used as the password for the new `AUTH
"internal connection" <password>` command.
- This API enables the modules to authenticate against other shards as
local connections.

## Notes on Behavior

- **ACL validation**:
- Commands dispatched by internal connections bypass ACL validation, to
give the caller full access regardless of the user with which it is
connected.

- **Command Visibility**:
- Internal commands **do not appear** in `COMMAND <subcommand>` and
`MONITOR` for non-internal connections.
- Internal commands **are logged** in the slow log, latency report and
commands' statistics to maintain observability.

- **`RM_Call()` Updates**:
  - **Non-internal connections**:
- Cannot execute internal commands when the command is sent with the `C`
flag (otherwise can).
- Internal connections bypass ACL validations (i.e., run as the
unrestricted user).

- **Internal commands' success**:
- Internal commands succeed upon being sent from either an internal
connection (i.e., authenticated via the new `AUTH "internal connection"
<internal_secret>` API), an AOF loading process, or from a master via
the replication link.
Any other connections that attempt to execute an internal command fail
with the `unknown command` error message raised.

- **`CLIENT LIST` flags**:
  - Added the `I` flag, to indicate that the connection is internal.

- **Lua Scripts**:
   - Prevented internal commands from being executed via Lua scripts.

---------

Co-authored-by: Meir Shpilraien <meir@redis.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes indication that this issue needs to be mentioned in the release notes

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants