Skip to content

SN panic in epoch handler on exit #3508

New issue
New issue
Closed
#3529
Closed
SN panic in epoch handler on exit#3508
#3529
Labels
I4No visible changesS4RoutineU2Seriously plannedbugSomething isn't workingneofs-storageStorage node application issues
Milestone
v0.49.0
@roman-khimov

Description

@roman-khimov
roman-khimov
opened on Aug 7, 2025

Expected Behavior

Send TERM, wait for node to shut down.

Current Behavior

Two nodes with #3507 problem gave this on restart:

Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612678 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 352371 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc01b5b8420?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc0108826c0?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612727 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 926546 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc022c78700?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc00a102060?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: error        neofs-node/netmap.go:241        could not make notary deposit        {"error": "could not calculate notary deposit: could not get notary balance: connection to the RPC node has been lost"}
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612796 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 543488 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc00dd8e840?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc02f79a570?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612792 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 636075 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xaa266a01d275ea23?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc02867cc00?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612821 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 1117485 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xaa266a01d275ea23?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc0320d6660?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612832 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 830851 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xaa266a01d275ea23?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc025c217d0?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612844 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 446749 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc0001c2070?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc027a57b30?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612864 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 734693 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xaa266a01d275ea23?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc03dc019b0?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612862 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 1022154 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc0085b8700?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc034d156e0?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c
Aug 07 06:05:19 metis3 neofs-node[2763445]: 2025/08/07 06:05:19.612724 [ants]: worker exits from panic: send on closed channel
Aug 07 06:05:19 metis3 neofs-node[2763445]: goroutine 1213193 [running]:
Aug 07 06:05:19 metis3 neofs-node[2763445]: runtime/debug.Stack()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/debug/stack.go:26 +0x5e
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:56 +0x85
Aug 07 06:05:19 metis3 neofs-node[2763445]: panic({0x105eda0?, 0x144d750?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         runtime/panic.go:792 +0x132
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch(0xc0001bc150, 0xc0017e6e88?)
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247 +0xfb
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.func1({0x1451800?, 0xc0084de210?})
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75 +0x1f
Aug 07 06:05:19 metis3 neofs-node[2763445]: github.com/panjf2000/ants/v2.(*goWorker).run.func1()
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:67 +0x8a
Aug 07 06:05:19 metis3 neofs-node[2763445]: created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 3871
Aug 07 06:05:19 metis3 neofs-node[2763445]:         github.com/panjf2000/ants/v2@v2.9.0/worker.go:48 +0x5c

It's certainly stuck and only KILL resolves the problem.

Possible Solution

Unknown.

Steps to Reproduce (for bugs)

Unknown.

Context

Mainnet.

Regression

Unknown.

Your Environment

  • Version used: 0.48.1
  • Server setup and configuration: mainnet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    I4No visible changesS4RoutineU2Seriously plannedbugSomething isn't workingneofs-storageStorage node application issues

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions