-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kv: NodeStatus keys not cleared on decommissioning #51304
Copy link
Copy link
Closed
Labels
A-kv-decom-rolling-restartDecommission and Rolling RestartsDecommission and Rolling RestartsA-kv-serverRelating to the KV-level RPC serverRelating to the KV-level RPC serverC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)S-3-ux-surpriseIssue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.
Description
We don't currently clear entries from "status-node-" keyspace when a node is removed from the cluster. This can cause confusion when performing aggregation over those keys. For instance, when working with crdb_internal.kv_store_status and aggregating over metrics->>'ranges.underreplicated'.
I don't know the full history or intention of these keys and the metrics they contain, but we should likely either:
- clear the metrics portion of the keys associated with decommissioned nodes
- delete the keys associated with decommissioned nodes entirely
cc. @tbg since you might be one of the few people who still knows how this subsystem works and what its intentions are.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-decom-rolling-restartDecommission and Rolling RestartsDecommission and Rolling RestartsA-kv-serverRelating to the KV-level RPC serverRelating to the KV-level RPC serverC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)S-3-ux-surpriseIssue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.Issue leaves users wondering whether CRDB is behaving properly. Likely to hurt reputation/adoption.