Skip to content

storage: support adding Stores to existing Nodes #39415

@rkruze

Description

@rkruze

Describe the problem

Please describe the issue you observed, and any steps we can take to reproduce it:

In an existing cluster, if I take one node down and then add a new store to the node, the node comes back up but it is not able to connect to the cluster.

To Reproduce

What did you do? Describe in your own words.

If possible, provide steps to reproduce the behavior:

  1. Setup a CRDB cluster with 3 nodes and 1 store each.
  2. Shutdown one CRDB node and then add a new store to this node and bring back up.
  3. Look at the logs files and notice the following:
W190807 17:27:33.231525 165 storage/store_rebalancer.go:227  [n1,s1,store-rebalancer] StorePool missing descriptor for local store
W190807 17:27:40.779085 35456 storage/replica_range_lease.go:982  [n1,replicate,s1,r24/1:/Table/5{6-7}] have been waiting 1m0s attempting to acquire lease
W190807 17:27:40.779149 35457 storage/replica_write.go:199  [n1,s1,r24/1:/Table/5{6-7}] have been waiting 60.00s for proposing command RequestLease [/Table/56,/Min).
This range is likely unavailable.
Please submit this message at

  https://github.com/cockroachdb/cockroach/issues/new/choose

along with

    https://yourhost:8080/#/reports/range/24

and the following Raft status: {"id":"1","term":9,"vote":"1","commit":31,"lead":"0","raftState":"StatePreCandidate","applied":31,"progress":{},"leadtransferee":"0"}

Expected behavior
The node should come back online and connect to the cluster with the new store.

Additional data / screenshots
If the problem is SQL-related, include a copy of the SQL query and the schema
of the supporting tables.

If a node in your cluster encountered a fatal error, supply the contents of the
log directories (at minimum of the affected node(s), but preferably all nodes).

Note that log files can contain confidential information. Please continue
creating this issue, but contact support@cockroachlabs.com to submit the log
files in private.

If applicable, add screenshots to help explain your problem.
Logs are here: cockroach.log
Environment:

  • CockroachDB version 19.1.3/19.1.2
  • Server OS: Linux and MacOS

Additional context
What was the impact?

Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

A-kvAnything in KV that doesn't belong in a more specific category.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.E-starterMight be suitable for a starter project for new employees or team members.S-3-productivitySevere issues that impede the productivity of CockroachDB developers.docs-donedocs-known-limitation

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions