-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: support adding Stores to existing Nodes #39415
Description
Describe the problem
Please describe the issue you observed, and any steps we can take to reproduce it:
In an existing cluster, if I take one node down and then add a new store to the node, the node comes back up but it is not able to connect to the cluster.
To Reproduce
What did you do? Describe in your own words.
If possible, provide steps to reproduce the behavior:
- Setup a CRDB cluster with 3 nodes and 1 store each.
- Shutdown one CRDB node and then add a new store to this node and bring back up.
- Look at the logs files and notice the following:
W190807 17:27:33.231525 165 storage/store_rebalancer.go:227 [n1,s1,store-rebalancer] StorePool missing descriptor for local store
W190807 17:27:40.779085 35456 storage/replica_range_lease.go:982 [n1,replicate,s1,r24/1:/Table/5{6-7}] have been waiting 1m0s attempting to acquire lease
W190807 17:27:40.779149 35457 storage/replica_write.go:199 [n1,s1,r24/1:/Table/5{6-7}] have been waiting 60.00s for proposing command RequestLease [/Table/56,/Min).
This range is likely unavailable.
Please submit this message at
https://github.com/cockroachdb/cockroach/issues/new/choose
along with
https://yourhost:8080/#/reports/range/24
and the following Raft status: {"id":"1","term":9,"vote":"1","commit":31,"lead":"0","raftState":"StatePreCandidate","applied":31,"progress":{},"leadtransferee":"0"}
Expected behavior
The node should come back online and connect to the cluster with the new store.
Additional data / screenshots
If the problem is SQL-related, include a copy of the SQL query and the schema
of the supporting tables.
If a node in your cluster encountered a fatal error, supply the contents of the
log directories (at minimum of the affected node(s), but preferably all nodes).
Note that log files can contain confidential information. Please continue
creating this issue, but contact support@cockroachlabs.com to submit the log
files in private.
If applicable, add screenshots to help explain your problem.
Logs are here: cockroach.log
Environment:
- CockroachDB version 19.1.3/19.1.2
- Server OS: Linux and MacOS
Additional context
What was the impact?
Add any other context about the problem here.