Add metrics for external RPC calls (success / error) #1524
Description
Currently zetaclientd seems to panic when the external RPC returns an error response. Exposing success/failure counts by network as a metric would be very useful for monitoring.
Panic details:
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","time":"2024-01-03T21:26:39Z","message":"starting core observer"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","module":"ZetaChainWatcher","time":"2024-01-03T21:26:39Z","message":"Starting Send Scheduler for zeta1euvlmpffq7gl77vlwlv97kn004zq6rdw0skdd3"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","module":"startup","time":"2024-01-03T21:26:39Z","message":"awaiting the os.Interrupt, syscall.SIGTERM signals..."}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"error","chain":"goerli_testnet","module":"WatchGasPrice","error":"400 Bad Request: {\"error\": \"request has an invalid host info\",\"id\":1}","time":"2024-01-03T21:26:39Z","message":"Err SuggestGasPrice:"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: block height: 3059406
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"error","chain":"goerli_testnet","module":"WatchGasPrice","time":"2024-01-03T21:26:39Z","message":"PostGasPrice error at zeta block : 3059406 "}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"checking tracker with hash :0xc378b09ea33bfa34b19ca198b45a51596d8d50efddc883887ed41714cfdde9fd and coin-type :Gas "}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Checking for all inTX : startBlock 44358778, toBlock 44358877"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx detected: 0xc378b09ea33bfa34b19ca198b45a51596d8d50efddc883887ed41714cfdde9fd, blocknum 44303774"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx value: 4000000000000000000"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx from: 0xe855E2dab793Dec569Dfad9edfc9f2EaBf9Acf6c"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Gas Deposit detected and reported: PostSend zeta tx: 6BBA4A83124CE5F2317BD78461761B7BFCB490495DF06D895C26BA8F15C619DB"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Vote submitted for inbound Tracker,Chain : mumbai_testnet,Ballot Identifier : 0xc9c563f1f49ab42550339492bc96b20a70dcb959fea008f74fe0de01ceff0c6f, coin-type Gas"}
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","module":"OutTxProcessorManager","time":"2024-01-03T21:26:42Z","message":"StartTryProcess 0x9c915eba4d041aeb78b930c909e4fb1f576ed358132dabc0921c587b96f29437-5-215827, numActiveProcessor 1"}
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2027c70]
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: goroutine 820 [running]:
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: github.com/zeta-chain/zetacore/zetaclient.(*CoreObserver).startSendScheduler(0x4001b60200)
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: /go/src/github.com/zeta-chain/node/zetaclient/zetacore_observer.go:245 +0xdf0
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: created by github.com/zeta-chain/zetacore/zetaclient.(*CoreObserver).MonitorCore
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: /go/src/github.com/zeta-chain/node/zetaclient/zetacore_observer.go:91 +0xe0
Jan 03 21:26:42 validator0-eu-west-1-testnet systemd[1]: zetaclientd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT