Summary

• Block the FSMEventType_RUN transition in Blockchain.SendFSMEvent whenever the local chain tip is below the network's highest hard-coded checkpoint. Catchup callers continue to log the rejection and re-enter CATCHINGBLOCKS, so the node stops bouncing between states while mid-IBD.
• Remove the unconditional Init-time RUN flip in blockvalidation.Server.Init (added in #3672). It bypassed the gate and was the load-bearing reason the node could lie about being RUNNING during IBD. Crash recovery from a stuck CATCHINGBLOCKS state is already covered by legacy startSync and restoreFSMState (both now subject to the gate).
• Networks with no checkpoints (regtest, brand-new chains) bypass the gate. Mainnet's highest is currently 938000, testnet3's is 1700000 (go-chaincfg.MainNetParams.Checkpoints / TestNet3Params.Checkpoints).

Why

Follow-up to #843. While investigating that ban incident on bsva-ovh-teranode-eu-3 (chain tip ~293k, mainnet highest checkpoint 938000), Coralogix showed 30 Sending Run event blockvalidation/Server (CATCHINGBLOCKS => Run) log lines in 2.5 hours, all from services/blockvalidation/catchup.go:973 restoreFSMState. The defer u.restoreFSMState(ctx, catchupCtx) at catchup.go:790 flips state back to RUN unconditionally on every catchup chunk exit, regardless of whether the chain is actually caught up to network tip. Each RUN window let the legacy service emit at least one P2SH-bearing tx inv to seed-mainnet-eu-1 and earn bad-txns-vout-p2sh BAN THRESHOLD EXCEEDED (the re-ban at 18:01:38 lines up exactly with the same-second RUN event).

Investigation also surfaced a second path: blockvalidation.Server.Init (added by PR #3672) checks at startup whether FSM is in CATCHINGBLOCKS and, if so, sends RUN with no consultation of chain height. Comment in the original PR: "this is probably a remnant of a crash". Intent was reasonable (don't pin the node in CATCHINGBLOCKS if the prior process crashed mid-catchup), but the implementation lies about progress whenever the chain genuinely is mid-IBD on restart. The right invariant is the one this PR enforces in SendFSMEvent.

Implementation

services/blockchain/Server.go:

• New guardRunBelowHighestCheckpoint(ctx) helper. Reads b.store.GetBestBlockHeader(ctx); compares meta height against blockchain.HighestCheckpointHeight(b.settings.ChainCfgParams.Checkpoints). Returns errors.NewStateError(...) when below.
• New exported HighestCheckpointHeight([]chaincfg.Checkpoint) uint32 helper. Replaces the private duplicate previously in services/blockvalidation/catchup.go.
• SendFSMEvent calls the guard for FSMEventType_RUN. The existing CATCHINGBLOCKS-cannot-be-manually-exited gate is preserved.

services/blockvalidation/Server.go:

• Remove the Init-time RUN flip (lines 527-541 before this PR). With the gate in place it was actively dangerous; without the gate it was the source of the eu-3 ban incident. Crash recovery for a stuck CATCHINGBLOCKS state still works through:
  • services/legacy/netsync/manager.go:546 startSync — issues RUN when a peer reports the same height as the local tip.
  • services/blockvalidation/catchup.go:973 restoreFSMState — defers RUN at the end of every catchup chunk.
    Both now go through the gate, so neither can fire while the local tip is below the highest checkpoint.

services/blockvalidation/catchup.go, services/blockvalidation/BlockValidation.go:

• Drop the private getHighestCheckpointHeight; rewire the three callsites (catchup.go:682, BlockValidation.go:1339, BlockValidation.go:1764) to the shared blockchain.HighestCheckpointHeight.

Caller behaviour for Run rejection:

• services/legacy/netsync/manager.go:546,1345,1973 — already log+continue.
• services/blockvalidation/catchup.go:973 — already log+continue.
• services/blockvalidation/Server.go — Init flip removed entirely.

No caller crashes on the new rejection; they just stay in CATCHINGBLOCKS until catchup genuinely passes the highest checkpoint.

Edge cases

• Networks with no checkpoints (regtest, brand-new chains): HighestCheckpointHeight returns 0 → guard returns nil → RUN allowed.
• Missing settings.ChainCfgParams: guard returns nil (defensive; production always populates).
• Store read error or nil meta: fails closed (StateError) so the node retries later rather than slipping into RUN with unknown chain state.

Test plan

services/blockchain/server_fsm_run_gate_test.go:

• TestHighestCheckpointHeight — nil/empty/unordered + MainNet (938000) + RegTest (0).

• TestGuardRunBelowHighestCheckpoint — table-driven: below/at/above checkpoint, no-checkpoint network permits, store error fails closed, nil meta fails closed.

• TestGuardRunBelowHighestCheckpoint_NilSettings — defensive paths.

• go build ./...

• go test -count=1 -race ./services/blockchain/ → 609 pass

• go test -count=1 -race ./services/blockvalidation/ → 562 pass

• go test -count=1 -race ./services/blockvalidation/ ./services/legacy/netsync/ → 605 pass (downstream callers unaffected)

• go vet ./services/blockchain/...

• Manual: starting testnet daemon from a mid-IBD chain state (height 1481322) no longer crashes Init; node comes up in CATCHINGBLOCKS and catchup runs.

Follow-ups

• The deeper issue (restoreFSMState issues RUN even when more catchup is needed) is mitigated by the gate but worth a separate change to make restoreFSMState consult "are we actually at tip?" before transitioning. Avoids the constant rejection log spam during IBD.
• Mainnet checkpoint list in go-chaincfg is currently stale (highest = 938000, real tip ~948677). Update the checkpoints periodically so the gate threshold tracks reality.
• Pair with fix(legacy): suppress tx relay while FSM != RUNNING #843 to fully close the SV-node ban vector: fix(legacy): suppress tx relay while FSM != RUNNING #843 stops outbound tx relay while not RUNNING; this PR stops bogus RUN transitions while below checkpoint.