Migrated from Jira: PM-21082
Content below preserved verbatim from the Jira ticket. Further updates should happen here.

Description

Situation: Last week we found that on performance environment (10 block producing nodes + 10 relays) one node (Alice) is not producing the blocks. It just acted as relay and skip it's turn for block production. We checked node logs, enabled AURA and sync debug logging - nothing.The node was started with the following parameters:

midnight-node --node-key some_key --chain chain-spec.json --base-path=./data --validator --reserved-only --reserved-nodes /dns/node1/tcp/30333/p2p/12D3Ko... --reserved-nodes /dns/node2/tcp/30333/p2p/12D3Ko... --listen-addr /ip4/0.0.0.0/tcp/30333 --keystore-path ./data/chains/midnight_perfnet/keystore --prometheus-port 9615 --prometheus-external

What we've done: After a few rounds of checking node logs, we also took a look at the folder , where the keys (AURA, GRANDPA, CROSSCHAIN) are placed: --keystore-path ./data/chains/midnight_perfnet/keystoreWe found that one of the keys (AURA) was incorrect. After adding the correct one - the Alice has started to produce blocks.

Possible improvements:

• Add INFO level log in case if there are no keys found in the keystore: AURA, GRANDPA, CROSSCHAIN

• Add INFO level log in to indicate whether current node is in the committee or not.

Original Jira metadata

Attachments

—

Comments from Jira

Comment by Oscar Bailey — 2026-01-05T15:03:30.054+0000

• There was no indication in logs that validator has issues with the keystore. Can we add logging for it?

If the validator's keys do not exist in the current committee, that’s not necessarily an error case - I would suggest a new info log instead:

On session change, the node could log whether or not it’s in the committee

Comment by Sinead Plunkett — 2026-01-05T16:26:41.877+0000

@@Oleksandr Romanov
Can you create a separate SRE Ticket for point2:

• We need to make sure that on environments (especially on Mainnet) - there is a dashboard and alert on chain density - so we will be notified if some nodes are not producing blocks.

Please triage it with @@Chris Ferry to ensure he’s aware of the timeline we need this completed for

Comment by Oleksandr Romanov — 2026-01-06T11:27:11.125+0000

@@Sinead Plunkett Sure. I added link task

Comment by Oleksandr Romanov — 2026-01-06T11:27:47.934+0000

@@Oscar Bailey I updated the required improvements in the description of the bug according to your suggestion

Comment by Oscar Bailey — 2026-01-06T11:34:57.727+0000

• Add INFO level log in case if there are no keys found in the keystore: AURA, GRANDPA, CROSSCHAIN

If there are no keys in the keystore, the node will crash on startup - so this is already covered

Comment by Sinead Plunkett — 2026-01-12T09:43:36.151+0000

@@Oscar Bailey Leaving this triaged as Medium

We have an SRE Ticket now for the Dashboard work - thank you @@Oleksandr Romanov

Dev Work will be to add as you suggested in your below Comment:

If the validator's keys do not exist in the current committee, that’s not necessarily an error case - I would suggest a new info log instead:

On session change, the node could log whether or not it’s in the committee

Comment by Sinead Plunkett — 2026-04-23T14:53:51.710+0100

From @@Darren Oliveiro-Priestnall

redact credentials and infrastructure details before publication.

CARDANO_SECURITY_PARAMETER='432' CARDANO_ACTIVE_SLOTS_COEFF='0.05' DB_SYNC_POSTGRES_CONNECTION_STRING='postgresql://postgres:password123@localhost:5432/cexplorer' FIRST_EPOCH_TIMESTAMP_MILLIS='1666656000000' EPOCH_DURATION_MILLIS='86400000' FIRST_EPOCH_NUMBER='0' FIRST_SLOT_NUMBER='0' BLOCK_STABILITY_MARGIN='0' midnight-node --node-key c8eab0efbea70ac81e96f87a75158d1f62ca132f37269f8a0329bc354e155a80 --chain chain-spec.json --base-path=./data --validator --reserved-only --reserved-nodes /dns/george.node.sc.iog.io/tcp/30333/p2p/12D3KooWHd2x2YkDSFCQkvSiSjDCbFeA6SGVokBpqjANbqaibqDB --reserved-nodes /dns/ferdie.node.sc.iog.io/tcp/30333/p2p/12D3KooWFjgK5tSFeu9JURw5GqooMopgerADwJoyi5uX3arEajE4 --listen-addr /ip4/0.0.0.0/tcp/30333 --keystore-path ./data/chains/midnight_perfnet/keystore --prometheus-port 9615 --prometheus-external 

Comment by Oleksandr Romanov — 2026-04-23T15:05:36.060+0100

@@Sinead Plunkett Removed unnecessary infra and key details.