Skip to content

Fixed panic in coordinator when not initialized#751

Merged
mattisonchao merged 1 commit intooxia-db:mainfrom
merlimat:fix-panic-coordinator
Oct 9, 2025
Merged

Fixed panic in coordinator when not initialized#751
mattisonchao merged 1 commit intooxia-db:mainfrom
merlimat:fix-panic-coordinator

Conversation

@merlimat
Copy link
Copy Markdown
Collaborator

@merlimat merlimat commented Oct 7, 2025

Coordinator is waiting for all storage nodes to come up the first time. The shard rebalancer is causing a panic after 30sec

Oct  7 12:28:23.311313 INF start shard rebalance avg-shard-ratio=+Inf component=load-balancer max-node-load-ratio=0 min-node-load-ratio=0 quarantine-nodes=[]
Oct  7 12:28:23.311867 INF end shard rebalance avg-shard-ratio=+Inf component=load-balancer max-node-load-ratio=0 min-node-load-ratio=0
Oct  7 12:28:23.312078 INF start leader rebalance component=load-balancer node-leaders={} quarantine-shards=[]
Oct  7 12:28:23.312220 INF end leader rebalance component=load-balancer
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x105f13b98]

goroutine 29 [running]:
github.com/emirpasic/gods/v2/lists/arraylist.(*List[...]).Size(...)
	/Users/mmerli/go/pkg/mod/github.com/emirpasic/gods/v2@v2.0.0-alpha.0.20250312000129-1d83d5ae39fb/lists/arraylist/arraylist.go:105
github.com/emirpasic/gods/v2/lists/arraylist.(*Iterator[...]).Next(...)
	/Users/mmerli/go/pkg/mod/github.com/emirpasic/gods/v2@v2.0.0-alpha.0.20250312000129-1d83d5ae39fb/lists/arraylist/iterator.go:28
github.com/oxia-db/oxia/coordinator/balancer.(*nodeBasedBalancer).rebalanceLeader(0x140001e98c0)
	/Users/mmerli/prg/oxia/coordinator/balancer/scheduler.go:394 +0x208
github.com/oxia-db/oxia/coordinator/balancer.(*nodeBasedBalancer).startBackgroundNotifier.func1()
	/Users/mmerli/prg/oxia/coordinator/balancer/scheduler.go:359 +0xc8
github.com/oxia-db/oxia/common/process.DoWithLabels.func1({0x106af1048?, 0x140001ca000?})
	/Users/mmerli/prg/oxia/common/process/pprof.go:46 +0x24
runtime/pprof.Do({0x106af1080?, 0x14000472370?}, {{0x140004a4020?, 0x1046f67e4?, 0x14000126ee8?}}, 0x140004f6f88)
	/Users/mmerli/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.darwin-arm64/src/runtime/pprof/runtime.go:51 +0x78
github.com/oxia-db/oxia/common/process.DoWithLabels({0x106af1080, 0x14000472370}, 0x14000162150, 0x1400028d0b0)
	/Users/mmerli/prg/oxia/common/process/pprof.go:42 +0x1b4
created by github.com/oxia-db/oxia/coordinator/balancer.(*nodeBasedBalancer).startBackgroundNotifier in goroutine 1
	/Users/mmerli/prg/oxia/coordinator/balancer/scheduler.go:348 +0x134

@mattisonchao mattisonchao merged commit bc3c5f6 into oxia-db:main Oct 9, 2025
5 checks passed
@merlimat merlimat deleted the fix-panic-coordinator branch October 9, 2025 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants