loqrecovery: use captured meta range content for LOQ plans by aliher1911 · Pull Request #94239 · cockroachdb/cockroach

aliher1911 · 2022-12-23T14:53:47Z

Note: only last commit belongs to this PR. Will update description once #93157 lands.

Previously loss of quorum recovery planner was using local replica info collected from all nodes to find source of truth for replicas that lost quorum.
With online approach local info snapshots don't have atomicity. This could cause planner to fail if available replicas are caught in different states on different nodes.
This commit adds alternative planning approach when range metadata is available. Instead of fixing individual replicas that can't make progress it finds ranges that can't make progress from metadata using descriptors and updates their replicas to recover from loss of quorum.
This commit also adds replica collection stage as a part of make-plan command itself. To invoke collection from a cluster instead of using files one needs to provide --host and other standard cluster connection related flags (--cert-dir, --insecure etc.) as appropriate.

Example command output for a local cluster with 3 out of 5 nodes surrvivng looks like:

~/tmp ❯❯❯ cockroach debug recover make-plan --insecure --host 127.0.0.1:26257 >recover-plan.json
Nodes scanned:           3
Total replicas analyzed: 228
Ranges without quorum:   15
Discarded live replicas: 0

Proposed changes:
  range r4:/System/tsd updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r80:/Table/106/1 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):3,(n4,s4):2].
  range r87:/Table/106/1/"paris"/"\xcc\xcc\xcc\xcc\xcc\xcc@\x00\x80\x00\x00\x00\x00\x00\x00(" updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):3,(n4,s4):2].
  range r88:/Table/106/1/"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x14" updating replica (n3,s3):3 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r105:/Table/106/1/"washington dc"/"L\xcc\xcc\xcc\xcc\xccL\x00\x80\x00\x00\x00\x00\x00\x00\x0f" updating replica (n3,s3):3 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n5,s5):1,(n4,s4):2].
  range r98:/Table/107/1/"boston"/"333333D\x00\x80\x00\x00\x00\x00\x00\x00\x03" updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r95:/Table/107/1/"seattle"/"ffffffH\x00\x80\x00\x00\x00\x00\x00\x00\x06" updating replica (n3,s3):2 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n4,s4):4,(n5,s5):3].
  range r125:/Table/107/1/"washington dc"/"DDDDDDD\x00\x80\x00\x00\x00\x00\x00\x00\x04" updating replica (n3,s3):2 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):1,(n5,s5):3].
  range r115:/Table/108/1/"boston"/"8Q\xeb\x85\x1e\xb8B\x00\x80\x00\x00\x00\x00\x00\x00n" updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r104:/Table/108/1/"new york"/"\x1c(\xf5\u008f\\I\x00\x80\x00\x00\x00\x00\x00\x007" updating replica (n2,s2):2 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):3].
  range r102:/Table/108/1/"seattle"/"p\xa3\xd7\n=pD\x00\x80\x00\x00\x00\x00\x00\x00\xdc" updating replica (n3,s3):2 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n4,s4):4,(n5,s5):3].
  range r126:/Table/108/1/"washington dc"/"Tz\xe1G\xae\x14L\x00\x80\x00\x00\x00\x00\x00\x00\xa5" updating replica (n3,s3):2 to (n3,s3):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):1,(n5,s5):3].
  range r86:/Table/108/3 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n4,s4):3,(n5,s5):2].
  range r59:/Table/109/1 updating replica (n2,s2):3 to (n2,s2):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].
  range r65:/Table/111/1 updating replica (n3,s3):3 to (n3,s3):15. Discarding available replicas: [], discarding dead replicas: [(n5,s5):4,(n4,s4):2].

Discovered dead nodes would be marked as decommissioned:
  n4, n5


Proceed with plan creation [y/N] y
Plan created.
To stage recovery application in half-online mode invoke:

'cockroach debug recover apply-plan  --host=127.0.0.1:26257 --insecure=true <plan file>'

Alternatively distribute plan to below nodes and invoke 'debug recover apply-plan --store=<store-dir> <plan file>' on:
- node n2, store(s) s2
- node n1, store(s) s1
- node n3, store(s) s3

Release note: None

Fixes: #93038
Fixes: #93046

cockroach-teamcity · 2022-12-23T14:53:55Z

This change is

erikgrinaker

Just minor stuff.

pkg/kv/kvserver/loqrecovery/utils.go

erikgrinaker · 2023-01-10T12:40:19Z

pkg/kv/kvserver/loqrecovery/utils.go

We allow use of Go generics now (#93735). This seems like a good opportunity to try it out with some generic map intersect/diff functions (or just use a library which provides it). Doesn't have to happen in this PR though, or at all.

erikgrinaker · 2023-01-10T12:43:31Z

pkg/cli/debug.go

Should we just get rid of dead-store-ids, to avoid the complexity of supporting both?

I think we should get rid of dead store ids in 23.2. Just to avoid any confusion on mixed cluster where old option which was the primary means disappears after upgrade. It should be less of an issue with 23.1 to 23.2 where we don't expect people to use dead-store-ids anymore.

pkg/cli/debug_recover_loss_of_quorum.go

pkg/kv/kvserver/loqrecovery/plan.go

pkg/kv/kvserver/loqrecovery/utils.go

pkg/kv/kvserver/loqrecovery/plan.go

Previously loss of quorum recovery planner was using local replica info collected from all nodes to find source of truth for replicas that lost quorum. With online approach local info snapshots don't have atomicity. This could cause planner to fail if available replicas are caught in different states on different nodes. This commit adds alternative planning approach when range metadata is available. Instead of fixing individual replicas that can't make progress it finds ranges that can't make progress from metadata using descriptors and updates their replicas to recover from loss of quorum. This commit also adds replica collection stage as a part of make-plan command itself. To invoke collection from a cluster instead of using files one needs to provide --host and other standard cluster connection related flags (--cert-dir, --insecure etc.) as appropriate. Release note: None

aliher1911 · 2023-01-17T18:31:06Z

bors r=erikgrinaker

craig · 2023-01-17T20:02:32Z

Build failed (retrying...):

Bazel Essential CI (Cockroach)

craig · 2023-01-17T22:50:05Z

Build succeeded:

Bazel Essential CI (Cockroach)

aliher1911 force-pushed the loq_05online_plan_replicas branch 4 times, most recently from 973507f to 70e07e9 Compare December 28, 2022 14:41

aliher1911 changed the title ~~loqrecovery: add replica info collection to admin server~~ loqrecovery: use captured meta range content for LOQ plans Dec 28, 2022

aliher1911 force-pushed the loq_05online_plan_replicas branch 5 times, most recently from 809abfb to 5305b54 Compare December 29, 2022 21:33

aliher1911 force-pushed the loq_05online_plan_replicas branch 10 times, most recently from c727406 to 36bace0 Compare January 9, 2023 13:07

aliher1911 marked this pull request as ready for review January 9, 2023 13:09

aliher1911 requested review from a team as code owners January 9, 2023 13:09

aliher1911 requested review from a team January 9, 2023 13:09

aliher1911 requested a review from a team as a code owner January 9, 2023 13:09

aliher1911 requested a review from erikgrinaker January 9, 2023 13:09

aliher1911 force-pushed the loq_05online_plan_replicas branch from 36bace0 to 8b8eb85 Compare January 10, 2023 12:59

erikgrinaker approved these changes Jan 10, 2023

View reviewed changes

aliher1911 force-pushed the loq_05online_plan_replicas branch from 8b8eb85 to 0618f19 Compare January 11, 2023 17:36

aliher1911 requested review from a team as code owners January 11, 2023 17:36

aliher1911 requested review from jayshrivastava and rhu713 and removed request for a team January 11, 2023 17:36

dhartunian removed the request for review from a team January 11, 2023 17:45

aliher1911 force-pushed the loq_05online_plan_replicas branch 8 times, most recently from 9b527a8 to f21dbac Compare January 16, 2023 16:24

aliher1911 force-pushed the loq_05online_plan_replicas branch from f21dbac to b5d5fc3 Compare January 17, 2023 17:05

craig bot merged commit 6fd5b04 into cockroachdb:master Jan 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

loqrecovery: use captured meta range content for LOQ plans#94239

loqrecovery: use captured meta range content for LOQ plans#94239
craig[bot] merged 1 commit intocockroachdb:masterfrom
aliher1911:loq_05online_plan_replicas

aliher1911 commented Dec 23, 2022 •

edited

Loading

Uh oh!

cockroach-teamcity commented Dec 23, 2022

Uh oh!

erikgrinaker left a comment

Uh oh!

Uh oh!

erikgrinaker Jan 10, 2023

Uh oh!

erikgrinaker Jan 10, 2023

Uh oh!

aliher1911 Jan 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aliher1911 commented Jan 17, 2023

Uh oh!

craig bot commented Jan 17, 2023

Uh oh!

craig bot commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aliher1911 commented Dec 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cockroach-teamcity commented Dec 23, 2022

Uh oh!

erikgrinaker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikgrinaker Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

erikgrinaker Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

aliher1911 Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aliher1911 commented Jan 17, 2023

Uh oh!

craig bot commented Jan 17, 2023

Uh oh!

craig bot commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aliher1911 commented Dec 23, 2022 •

edited

Loading