-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: allocate NodeID via one-off RPC to initialized node #32574
Description
Our handling of the NodeID is currently quite involved (and leaks into many places) because we pretty much require a working node to be set up before it can use its own KV store to request a NodeID.
There are two levels of improvement here. The first is that once Gossip is connected, and the Node is uninitialized, we use a one-off RPC AllocateNodeID which simply asks a remote node of our choosing (taken from Gossip) to allocate a NodeID for us. This shifts the complexity of using the KV store to a node that has it in perfect working condition. (Similar improvements apply to StoreID allocation, which we may be able to hoist out of the depths of (*Node).Start where it is buried right now).
This removes uninitialized NodeIDs from everywhere except Gossip. Removing it from Gossip is more involved, depending on what guarantees we want to offer. The basic idea is to read the Gossip bootstrap list directly to send AllocateNodeID, and not starting Gossip before that has succeeded. However, consider the case in which a node isn't able to reach out to other nodes (perhaps its join flags are stale) but other nodes are able to reach out to this node (this is a poor setup, but it could happen): we'd get stuck, unless there's at least a minimal subset of Gossip running that would add the peer's address to the list of addresses we pick from for AllocateNodeID. This doesn't seem terrible to implement, but it's a little more involved.
Related to #30553.