storage: endless spam from split queue retries #38183
Description
When doing various operations on a local cluster, I've gotten endless rapid fire spam from the split queue that looks like this:
I190616 22:01:16.651467 303148 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.656905 303151 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56656] (zone config)
I190616 22:01:16.663793 303151 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.669182 303154 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56657] (zone config)
I190616 22:01:16.676599 303154 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.682135 301143 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56658] (zone config)
I190616 22:01:16.689591 301143 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.694930 302860 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56659] (zone config)
I190616 22:01:16.701739 302860 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.707172 302863 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56660] (zone config)
I190616 22:01:16.714354 302863 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.719807 302866 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56661] (zone config)
I190616 22:01:16.726593 302866 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.731990 301147 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56662] (zone config)
I190616 22:01:16.738716 301147 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.744122 303159 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56663] (zone config)
I190616 22:01:16.750865 303159 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.756206 303162 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56664] (zone config)
I190616 22:01:16.763056 303162 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.768365 301151 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56665] (zone config)
I190616 22:01:16.776631 301151 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.782099 303205 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56666] (zone config)
I190616 22:01:16.789057 303205 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.794444 299042 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56667] (zone config)
I190616 22:01:16.801219 299042 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
I190616 22:01:16.806649 302414 storage/replica_command.go:290 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] initiating a split of this range at key /Table/3871 [r56668] (zone config)
I190616 22:01:16.813199 302414 storage/split_queue.go:146 [n1,split,s1,r8613/1:/{Table/3870/2…-Max}] split saw concurrent descriptor modification; maybe retrying
It doesn't cease until I restart the cluster. @tbg could you triage this? Let me know if you need help reproing - I haven't tried to make a small repro case yet because I thought the core team might understand what's going on quickly.
Jira issue: CRDB-10827