At the moment, most of the PrivKey methods (Sign, PubKey) don't have a way to return errors, so we must panic if we can't query the Ledger. This will also be a problem with future HW wallets or any other kind of asynchronous signatures.

Changing these interfaces might be annoying downstream though.