RackAwareRoundRobinPolicy still send out requests to other racks #255
Description
I have use case with 6 nodes, and c-s is using RackAwareRoundRobinPolicy that targets only rack 1a
and all stress commands are showing the following logs
Connected to cluster: longevity-cdc-100gb-4h-rackawar-db-cluster-7f55814f, max pending requests per connection null, max connections per host 8
WARN 17:17:34,494 Some contact points don't match local rack. Local rack = 1a. Non-conforming contact points: /10.4.8.52:9042 (dc=eu-west, rack=1c),/10.4.6.128:9042 (dc=eu-west, rack=1b),/10.4.10.155:9042 (dc=eu-west, rack=1c),/10.4.6.92:9042 (dc=eu-west, rack=1b)
Datatacenter: eu-west; Host: /10.4.2.35; Rack: 1a
Datatacenter: eu-west; Host: /10.4.8.52; Rack: 1c
Datatacenter: eu-west; Host: /10.4.6.128; Rack: 1b
Datatacenter: eu-west; Host: /10.4.10.155; Rack: 1c
Datatacenter: eu-west; Host: /10.4.6.92; Rack: 1b
Datatacenter: eu-west; Host: /10.4.3.54; Rack: 1a
Created schema. Sleeping 6s for propagation.
I assume that regredless of which nodes are passed as contact points, the driver would still work base on the logic of RackAwareRoundRobinPolicy and communicate only the relevant rack
but other nodes in rack 1b, 1c, are getting requests
Installation details
Kernel Version: 5.15.0-1049-aws
Scylla version (or git commit hash): 5.5.0~dev-20231027.227136ddf54f with build-id 420b5e5ef0c99c93991034185a1061ec90ab2c43
Cluster size: 6 nodes (i4i.4xlarge)
Scylla Nodes used in this run:
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-7 (3.249.58.125 | 10.4.10.54) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-6 (54.155.166.212 | 10.4.8.52) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-5 (34.240.100.209 | 10.4.10.155) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-4 (34.254.65.243 | 10.4.6.92) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-3 (54.72.53.223 | 10.4.6.128) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-2 (63.35.252.244 | 10.4.2.35) (shards: 14)
- longevity-cdc-100gb-4h-rackawar-db-node-7f55814f-1 (18.203.234.95 | 10.4.3.54) (shards: 14)
OS / Image: ami-0445669e3c59c66ab (aws: undefined_region)
Test: longevity-cdc-100gb-4h-test
Test id: 7f55814f-0d8c-4707-b94c-a05efbf2b990
Test name: scylla-staging/fruch/longevity-cdc-100gb-4h-test
Test config file(s):
Logs and commands
- Restore Monitor Stack command:
$ hydra investigate show-monitor 7f55814f-0d8c-4707-b94c-a05efbf2b990 - Restore monitor on AWS instance using Jenkins job
- Show all stored logs command:
$ hydra investigate show-logs 7f55814f-0d8c-4707-b94c-a05efbf2b990
Logs:
- db-cluster-7f55814f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/db-cluster-7f55814f.tar.gz
- sct-runner-events-7f55814f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/sct-runner-events-7f55814f.tar.gz
- sct-7f55814f.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/sct-7f55814f.log.tar.gz
- loader-set-7f55814f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/loader-set-7f55814f.tar.gz
- monitor-set-7f55814f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/monitor-set-7f55814f.tar.gz
- parallel-timelines-report-7f55814f.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7f55814f-0d8c-4707-b94c-a05efbf2b990/20231102_204619/parallel-timelines-report-7f55814f.tar.gz