Skip to content

tests: add test for rebalancer thrashing#9371

Merged
cuongdo merged 1 commit intocockroachdb:masterfrom
cuongdo:rebalancer_thrashing_test
Sep 15, 2016
Merged

tests: add test for rebalancer thrashing#9371
cuongdo merged 1 commit intocockroachdb:masterfrom
cuongdo:rebalancer_thrashing_test

Conversation

@cuongdo
Copy link
Copy Markdown
Contributor

@cuongdo cuongdo commented Sep 14, 2016

Add a test to ensure that we don't try to rebalance when the cluster is
sufficiently balanced.


This change is Reviewable

a.options.Deterministic = true
defer stopper.Stop()

stores := []*roachpb.StoreDescriptor{
Copy link
Copy Markdown
Contributor

@tamird tamird Sep 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the fact that there are 4 stores coupled to the fact that there are 4 ranges in the factory functions below? if not, better to make the numbers different to avoid chekhov's gun.

// a range count that's above the target range count for the rebalancer, so it
// should be rebalanced from.
oneReplicaAboveRebalanceTarget := func(mean int32) [4]testReplica {
var replicas [4]testReplica
Copy link
Copy Markdown
Contributor

@tamird tamird Sep 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps type testCase [4]testReplica to avoid repeating yourself.

@petermattis
Copy link
Copy Markdown
Collaborator

Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful.


storage/allocator_test.go, line 529 at r1 (raw file):

      // test the case where 3 nodes are above the mean but below the rebalancer's
      // target range count but one node is well below the mean. We still want to
      // rebalance to severely underused stores.

Can you add that scenario here, verify it fails and then comment it out?


Comments from Reviewable

@petermattis
Copy link
Copy Markdown
Collaborator

:lgtm:


Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful.


Comments from Reviewable

@cuongdo cuongdo force-pushed the rebalancer_thrashing_test branch from 3d81e9c to 0823281 Compare September 14, 2016 20:15
@cuongdo
Copy link
Copy Markdown
Contributor Author

cuongdo commented Sep 14, 2016

Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions, some commit checks pending.


storage/allocator_test.go, line 469 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

is the fact that there are 4 stores coupled to the fact that there are 4 ranges in the factory functions below? if not, better to make the numbers different to avoid chekhov's gun.

rewrote substantially to make the intent clearer

storage/allocator_test.go, line 501 at r1 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

perhaps type testCase [4]testReplica to avoid repeating yourself.

no longer relevant

storage/allocator_test.go, line 529 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Can you add that scenario here, verify it fails and then comment it out?

added

Comments from Reviewable

@tamird
Copy link
Copy Markdown
Contributor

tamird commented Sep 14, 2016

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 6 unresolved discussions, some commit checks pending.


storage/allocator_test.go, line 475 at r2 (raw file):

  // should be rebalanced from.
  oneStoreAboveRebalanceTarget := func(mean int32, numStores int) []testStore {
      if minStores := 4; numStores < minStores {

why 4? extract a constant to share with oneUnderusedStore and anchor a comment on it.


storage/allocator_test.go, line 482 at r2 (raw file):

          stores[i].rangeCount = mean
      }
      thresholdDelta := int32(float64(mean)*rebalanceThreshold + 1)

perhaps surplus for consistency with deficit below? the math here seems sketchy, too, in that unlike below, the first and last replicas will be exact mirrors. i thought the intention was to have one outlier store.


storage/allocator_test.go, line 489 at r2 (raw file):

  }

  // Returns a slice of stores with the specified mean (fill in TBD)

Looks like you forgot to flesh out this comment.


storage/allocator_test.go, line 494 at r2 (raw file):

          t.Fatalf("# of stores %d < min %d", numStores, minStores)
      }
      stores := make([]testStore, 10)

s/10/numStores/?


storage/allocator_test.go, line 498 at r2 (raw file):

          stores[i].rangeCount = mean
      }
      // Subtract a large number of ranges from the first store and add that delta

better if you clarify "large". it seems to be the minimum number that will trigger the rebalancer's heuristic?


Comments from Reviewable

Add a test to ensure that we don't try to rebalance when the cluster is
sufficiently balanced.
@cuongdo cuongdo force-pushed the rebalancer_thrashing_test branch from 0823281 to 8af068f Compare September 14, 2016 21:01
@cuongdo
Copy link
Copy Markdown
Contributor Author

cuongdo commented Sep 14, 2016

Review status: 0 of 1 files reviewed at latest revision, 8 unresolved discussions.


storage/allocator_test.go, line 475 at r2 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

why 4? extract a constant to share with oneUnderusedStore and anchor a comment on it.

Done

storage/allocator_test.go, line 482 at r2 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

perhaps surplus for consistency with deficit below? the math here seems sketchy, too, in that unlike below, the first and last replicas will be exact mirrors. i thought the intention was to have one outlier store.

Renamed. I've made this function more consistent with the other for readability, but functionally, it's a moot point until we implement whatever heuristic we choose to decide if a replica doesn't have enough ranges.

storage/allocator_test.go, line 489 at r2 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

Looks like you forgot to flesh out this comment.

Done.

storage/allocator_test.go, line 494 at r2 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

s/10/numStores/?

Done.

storage/allocator_test.go, line 498 at r2 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

better if you clarify "large". it seems to be the minimum number that will trigger the rebalancer's heuristic?

Done.

Comments from Reviewable

@tamird
Copy link
Copy Markdown
Contributor

tamird commented Sep 14, 2016

:lgtm:


Reviewed 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


Comments from Reviewable

@cuongdo cuongdo merged commit d310d95 into cockroachdb:master Sep 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants