redis-cli: Do DNS lookup before sending CLUSTER MEET #10436

zuiderkwast · 2022-03-17T08:54:03Z

Affects --cluster create and --cluster add-node.

There are no tests added. (Is it possible without setting up DNS records? Can we create a hosts file in github actions?)

@liuchong can you test this with your nodes? (redis-cli --cluster create redis-node-1:7001 redis-node-2:7002 redis-node-3:7003 redis-node-4:7004 redis-node-5:7005 redis-node-6:7006 --cluster-replicas 1)

liuchong · 2022-03-17T12:13:47Z

Yes, let me try! 🚀

liuchong · 2022-03-18T03:22:10Z

It works, thanks! 🎉

Seems a new --protected-mode added, when I create cluster it reported an error. after I added a --protedted-mode no the error disappeared and cluster created.

Here is the logs:

redis-cli --cluster create redis-node-1:7001 redis-node-2:7002 redis-node-3:7003 redis-node-4:7004 redis-node-5:7005 redis-node-6:7006 --cluster-replicas 1
[ERR] Node redis-node-1:7001 DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a an authentication password for the default user. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

redis-cli --cluster create redis-node-1:7001 redis-node-2:7002 redis-node-3:7003 redis-node-4:7004 redis-node-5:7005 redis-node-6:7006 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-node-5:7005 to redis-node-1:7001
Adding replica redis-node-6:7006 to redis-node-2:7002
Adding replica redis-node-4:7004 to redis-node-3:7003
M: e84a70c4687fb4e52f8639b73d1acdc422aa5fad redis-node-1:7001
   slots:[0-5460] (5461 slots) master
M: 8ac5d42994b43b868ff0022d2ab437921e916322 redis-node-2:7002
   slots:[5461-10922] (5462 slots) master
M: 2faede35575d281c44290fbfeb884faeaef04507 redis-node-3:7003
   slots:[10923-16383] (5461 slots) master
S: 09bc6cbf6ec6e7f0f4d9011a6f34a93e2c9f2559 redis-node-4:7004
   replicates 2faede35575d281c44290fbfeb884faeaef04507
S: 4a470ad16002fdc943d9799975314281da6b5dec redis-node-5:7005
   replicates e84a70c4687fb4e52f8639b73d1acdc422aa5fad
S: a6e99146d7fc6f665019bc9827ad34d6421fbdd8 redis-node-6:7006
   replicates 8ac5d42994b43b868ff0022d2ab437921e916322
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node redis-node-1:7001)
M: e84a70c4687fb4e52f8639b73d1acdc422aa5fad redis-node-1:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: a6e99146d7fc6f665019bc9827ad34d6421fbdd8 172.18.0.5:7006
   slots: (0 slots) slave
   replicates 8ac5d42994b43b868ff0022d2ab437921e916322
M: 2faede35575d281c44290fbfeb884faeaef04507 172.18.0.3:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 8ac5d42994b43b868ff0022d2ab437921e916322 172.18.0.8:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 09bc6cbf6ec6e7f0f4d9011a6f34a93e2c9f2559 172.18.0.2:7004
   slots: (0 slots) slave
   replicates 2faede35575d281c44290fbfeb884faeaef04507
S: 4a470ad16002fdc943d9799975314281da6b5dec 172.18.0.4:7005
   slots: (0 slots) slave
   replicates e84a70c4687fb4e52f8639b73d1acdc422aa5fad
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

liuchong · 2022-03-18T03:43:23Z

Do some test:

redis-cli -h redis-node-1 -c -p 7001
redis-node-1:7001> keys *
(empty array)
redis-node-1:7001> set x 1
-> Redirected to slot [16287] located at 172.18.0.3:7003
OK
172.18.0.3:7003> set y 1
OK
172.18.0.3:7003> set z 1
-> Redirected to slot [8157] located at 172.18.0.8:7002
OK
172.18.0.8:7002> set x 1
-> Redirected to slot [16287] located at 172.18.0.3:7003
OK
172.18.0.3:7003> set y 2
OK
172.18.0.3:7003> set z 3
-> Redirected to slot [8157] located at 172.18.0.8:7002
OK

zuiderkwast · 2022-03-18T06:33:44Z

Great! Thanks for testing! Do you want to try --cluster add-node too? It is also affected. It's using CLUSTER MEET and does DNS lookup in the same way.

liuchong · 2022-03-18T10:04:09Z

It seems also works 👍

Here are the logs:

redis-cli --cluster add-node redis-node-7:7007 redis-node-3:7003
>>> Adding node redis-node-7:7007 to cluster redis-node-3:7003
>>> Performing Cluster Check (using node redis-node-3:7003)
M: 2faede35575d281c44290fbfeb884faeaef04507 redis-node-3:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: a6e99146d7fc6f665019bc9827ad34d6421fbdd8 172.18.0.9:7006
   slots: (0 slots) slave
   replicates 8ac5d42994b43b868ff0022d2ab437921e916322
M: 8ac5d42994b43b868ff0022d2ab437921e916322 172.18.0.7:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: e84a70c4687fb4e52f8639b73d1acdc422aa5fad 172.18.0.4:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4a470ad16002fdc943d9799975314281da6b5dec 172.18.0.8:7005
   slots: (0 slots) slave
   replicates e84a70c4687fb4e52f8639b73d1acdc422aa5fad
S: 09bc6cbf6ec6e7f0f4d9011a6f34a93e2c9f2559 172.18.0.6:7004
   slots: (0 slots) slave
   replicates 2faede35575d281c44290fbfeb884faeaef04507
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to redis-node-7:7007 to verify there is no functions in it
>>> Send FUNCTION RESTORE to redis-node-7:7007
>>> Send CLUSTER MEET to node redis-node-7:7007 to make it join the cluster.
[OK] New node added correctly.

redis-cli -h redis-node-7 -c -p 7007
redis-node-7:7007> set m 7
-> Redirected to slot [15627] located at 172.18.0.3:7003
OK
172.18.0.3:7003> set n 8
-> Redirected to slot [3432] located at 172.18.0.4:7001
OK
172.18.0.4:7001> get n
"8"
172.18.0.4:7001> get m
-> Redirected to slot [15627] located at 172.18.0.3:7003
"7"
172.18.0.3:7003>

redis-cli -h redis-node-2 -c -p 7002
redis-node-2:7002> get m
-> Redirected to slot [15627] located at 172.18.0.3:7003
"7"
172.18.0.3:7003> get n
-> Redirected to slot [3432] located at 172.18.0.4:7001
"8"
172.18.0.4:7001> get x
-> Redirected to slot [16287] located at 172.18.0.3:7003
"1"
172.18.0.3:7003> get y
"2"
172.18.0.3:7003> get z
-> Redirected to slot [8157] located at 172.18.0.7:7002
"3"
172.18.0.7:7002>

yossigo

@zuiderkwast Maybe use localhost for a test? I think we can assume that it will always work.

liuchong · 2022-03-20T09:11:37Z

It seems localhost also works.

I tested create cluster and add node and succeeded.
Because I got some problems when I compile it on macbook directly, I was using a docker command like this:

# In macos
docker run --rm -v $PWD:$PWD -w $PWD -it builder-common bash
# In docker
make
cd 700x && redis-server redis.conf &

conf:

port 7000 # 7000..7006
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

These are the succeeded logs:

redis-cli --cluster create localhost:7000 localhost:7001 localhost:7002 localhost:7003 localhost:7004 localhost:7005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica localhost:7004 to localhost:7000
Adding replica localhost:7005 to localhost:7001
Adding replica localhost:7003 to localhost:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: e294c13fe45aa07ed3b2010cc76df81f6ddb8989 localhost:7000
   slots:[0-5460] (5461 slots) master
M: 4391871060a683d60767c7a9a1d0d15cce4b779c localhost:7001
   slots:[5461-10922] (5462 slots) master
M: cc26c26d673edf0740b4959b69496e043fd90884 localhost:7002
   slots:[10923-16383] (5461 slots) master
S: d9617ec823876c1489aeab35d4f4739205c22a5a localhost:7003
   replicates e294c13fe45aa07ed3b2010cc76df81f6ddb8989
S: 77fd5175910eaadaf0b58f44f14dad0bad8938e0 localhost:7004
   replicates 4391871060a683d60767c7a9a1d0d15cce4b779c
S: 73c6291f888c6687844e4023389360ad5b9fa5a6 localhost:7005
   replicates cc26c26d673edf0740b4959b69496e043fd90884
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node localhost:7000)
M: e294c13fe45aa07ed3b2010cc76df81f6ddb8989 localhost:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 73c6291f888c6687844e4023389360ad5b9fa5a6 127.0.0.1:7005
   slots: (0 slots) slave
   replicates cc26c26d673edf0740b4959b69496e043fd90884
S: d9617ec823876c1489aeab35d4f4739205c22a5a 127.0.0.1:7003
   slots: (0 slots) slave
   replicates e294c13fe45aa07ed3b2010cc76df81f6ddb8989
M: cc26c26d673edf0740b4959b69496e043fd90884 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 4391871060a683d60767c7a9a1d0d15cce4b779c 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 77fd5175910eaadaf0b58f44f14dad0bad8938e0 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 4391871060a683d60767c7a9a1d0d15cce4b779c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli --cluster add-node localhost:7006 localhost:7005
>>> Adding node localhost:7006 to cluster localhost:7005
>>> Performing Cluster Check (using node localhost:7005)
S: 73c6291f888c6687844e4023389360ad5b9fa5a6 localhost:7005
   slots: (0 slots) slave
   replicates cc26c26d673edf0740b4959b69496e043fd90884
M: cc26c26d673edf0740b4959b69496e043fd90884 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: e294c13fe45aa07ed3b2010cc76df81f6ddb8989 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 4391871060a683d60767c7a9a1d0d15cce4b779c 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: d9617ec823876c1489aeab35d4f4739205c22a5a 127.0.0.1:7003
   slots: (0 slots) slave
   replicates e294c13fe45aa07ed3b2010cc76df81f6ddb8989
S: 77fd5175910eaadaf0b58f44f14dad0bad8938e0 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 4391871060a683d60767c7a9a1d0d15cce4b779c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to localhost:7006 to verify there is no functions in it
>>> Send FUNCTION RESTORE to localhost:7006
>>> Send CLUSTER MEET to node localhost:7006 to make it join the cluster.
[OK] New node added correctly.

zuiderkwast · 2022-03-21T22:03:18Z

I tried putting localhost in various places in the hostname cluster test suite. On my laptop I get ::1 for localhost and then it hangs somewhere. Perhaps we should mimick what hiredis does, which is to prefer IPv4 over IPv6...

zuiderkwast · 2022-03-28T10:17:13Z

@zuiderkwast Maybe use localhost for a test? I think we can assume that it will always work.

@yossigo Done. Also changed to prefer IPv4 over IPv6 (new flag for anetResolve) which is what hiredis does.

yossigo · 2022-03-28T10:21:34Z

@zuiderkwast
The reason it fails is because of the bind 127.0.0.1 we have as a default config. I actually don't like this hard-coded preference of IPv4, that's what /etc/gai.conf is there to do in a configurable way.

There are other options to fix the tests:

Modify the bind parameter to match localhost resolution
Use HOSTALIASES env variable and create a local aliases file (I suppose it's Linux/glibc only)

But I'm not sure if any of that is worth the trouble, I think I'd just go with the previous implementation and no test.

This reverts commit 49055c3.

zuiderkwast · 2022-03-28T14:44:12Z

@yossigo If we bind to ::1 instead of 127.0.0.1, there are a lot of tests which fail, probably becuase they're hard-coded to 127.0.0.1. If we bind to both, tests fail too (not sure why but I guess the test machinery takes the value to be a single IP address).

Regarding HOSTALIASES: The file maps alias names to canonical host names, but the canonical name must be resolvable IIUC. I don't think you can specify an IP address as the target.

I have reverted to the original implementation without tests, as suggested.

zuiderkwast · 2022-03-29T02:01:20Z

But in fact:

redis-cli -p 7001
127.0.0.1:7001> set foo bar
(error) MOVED 12182 127.0.0.1:7003

@liuchong You need to use -c to make redis-cli follow redirects. Does that solve your problem?

liuchong · 2022-03-30T04:07:57Z

But in fact:
redis-cli -p 7001
127.0.0.1:7001> set foo bar
(error) MOVED 12182 127.0.0.1:7003
@liuchong You need to use -c to make redis-cli follow redirects. Does that solve your problem?

Yes, the problem is solved, thanks! Forgot the -c 😅

This bug was introduced in redis#10344 (7.0.3), and it breaks the redis-cli --cluster create usage in redis#10436 (7.0 RC3). At the same time, the cluster-port support introduced in redis#10344 cannot use the DNS lookup brought by redis#10436.

…node (#11151) This bug was introduced in #10344 (7.0.3), and it breaks the redis-cli --cluster create usage in #10436 (7.0 RC3). At the same time, the cluster-port support introduced in #10344 cannot use the DNS lookup brought by #10436.

…node (#11151) This bug was introduced in #10344 (7.0.3), and it breaks the redis-cli --cluster create usage in #10436 (7.0 RC3). At the same time, the cluster-port support introduced in #10344 cannot use the DNS lookup brought by #10436. (cherry picked from commit c2b0c13)

…node (redis#11151) This bug was introduced in redis#10344 (7.0.3), and it breaks the redis-cli --cluster create usage in redis#10436 (7.0 RC3). At the same time, the cluster-port support introduced in redis#10344 cannot use the DNS lookup brought by redis#10436.

redis-cli: Do DNS lookup before sending CLUSTER MEET

28641f2

liuchong approved these changes Mar 18, 2022

View reviewed changes

zuiderkwast marked this pull request as ready for review March 18, 2022 06:34

oranagra requested a review from yossigo March 18, 2022 12:08

yossigo reviewed Mar 18, 2022

View reviewed changes

oranagra added the 7.0-must-have label Mar 27, 2022

oranagra mentioned this pull request Mar 27, 2022

Clusterbus extensions and hostname support #9530

Merged

3 tasks

Prefer IPv4 over IPv6 (like hiredis) and use localhost in tests

49055c3

Revert "Prefer IPv4 over IPv6 (like hiredis) and use localhost in tests"

2ae3c59

This reverts commit 49055c3.

yossigo added the release-notes indication that this issue needs to be mentioned in the release notes label Mar 29, 2022

yossigo approved these changes Mar 29, 2022

View reviewed changes

yossigo merged commit 35bb021 into redis:unstable Mar 29, 2022

zuiderkwast deleted the redis-cli-dns-lookup-for-cluster-meet branch March 30, 2022 08:08

oranagra mentioned this pull request Apr 5, 2022

Release 7.0 rc3 #10532

Merged

outsinre mentioned this pull request Aug 18, 2022

[BUG] cannot create cluster with docker compose service name #11148

Closed

enjoy-binbin mentioned this pull request Aug 19, 2022

Fix Invalid node address specified in redis-cli --cluster create/add-node #11151

Merged

zuiderkwast mentioned this pull request Sep 1, 2022

Resolve IPv6 address takes too much time redis/hiredis#696

Closed

kjoe mentioned this pull request Mar 23, 2024

Redis cluster returns IP address of master which not exist #4289

Open

redis-cli: Do DNS lookup before sending CLUSTER MEET #10436

redis-cli: Do DNS lookup before sending CLUSTER MEET #10436

Uh oh!

Conversation

zuiderkwast commented Mar 17, 2022

Uh oh!

liuchong commented Mar 17, 2022

Uh oh!

liuchong commented Mar 18, 2022

Uh oh!

liuchong commented Mar 18, 2022

Uh oh!

zuiderkwast commented Mar 18, 2022

Uh oh!

liuchong commented Mar 18, 2022

Uh oh!

yossigo left a comment

Choose a reason for hiding this comment

Uh oh!

liuchong commented Mar 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zuiderkwast commented Mar 21, 2022

Uh oh!

zuiderkwast commented Mar 28, 2022

Uh oh!

yossigo commented Mar 28, 2022

Uh oh!

zuiderkwast commented Mar 28, 2022

Uh oh!

zuiderkwast commented Mar 29, 2022

Uh oh!

liuchong commented Mar 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liuchong commented Mar 20, 2022 •

edited

Loading

liuchong commented Mar 30, 2022 •

edited

Loading