This issue #1790 made me to come up with an idea of a diagnostic runtime.

To recap: In contracts we have the lack of debuggability problem. In the best case, if the top-level contract fails you get the execution failure error. In the worst case, because contract failures don't cascade by default (i.e. if a leaf contract call fails it doesn't propagate to the root call) you don't get anything. There are even no way to log a message.

The simplest way of solving it is by adding some little features such as posting an event every time a contract generates an error, or e.g. add a special function that logs message that is available only for a testnet.

The problem with a testnet only debugging capabilities that they are no use for the production nets. Mixing debugging code into the runtime is also not the best idea either, because it adds overhead, possibly pessimizes the normal path, increases binary size and enlarges security surface.

It seems to me that a better solution would be to provide this functionality off chain.

Enter diagnostic runtime.

Imagine we had:

1. A special cfg feature in a runtime, which enables some off-chain debugging capabilities.
2. An RPC method that is the same as state_call, but that takes one extra parameter: a file path to a wasm runtime to use to execute.
3. Potentially, a way for a runtime to output a large chunk of data that can be returned by the aforementioned RPC method. Possibly in a file.

Having this, we could add a special path (behind a cfg feature) in the contract module logic which output a trace information.

This, for example, can be done exclusively on the runtime level without touching the substrate host. Here are a few ideas:

1. We can log traps in the trace.
2. We can record calls/instantiations, their parameters such as transferred value, consumed gas, input/output buffers.
3. At the present, the contract module performs instrumentation before executing a contract (e.g. to inject gas metering statements). We could alter the instrumentation so as to add extra statements and tracing. Here are few examples:
   1. We can insert a call to an ext_begin_fn function in the prologue and a call to an ext_end_fn in the epilogue of each function. With this we can construct stack traces inside of contracts in the case of errors.
   2. Or could can instrument every execution path to
   3. We could log every parameter
   4. In the extreme, we could instrument every instruction and get every operand trace.
   5. or as an another extreme option, we could alter wasmi and compile it in wasm and use it instead of using sandboxing.
4. we can add a function ext_print to the contract runtime. In the normal path it just doesn't do anything (i.e. don't even charge for gas for reading the code), but if the diagnostic feature is enabled the function reads the given buffer and records it in the trace. I think @Robbepop could be interested in this one.

After the execution we can assemble this as a trace and spew it out somehow, which could be decoded by tools (or alternatively, we could use human-readable format) or block explorers.

This setup gives us sheer number of possibilities for diagnostics without burdening the substrate host APIs. There is a caveat though: diagnostic runtimes likely would want to be state-compatible (i.e. produce the same state root) with the on-chain runtime just for the sake of being exchangable (i.e. for doing execute_block) and that might be tricky in one cases and limiting in other. However, this is not a hard requirement.

I think it might be possible that this mechanism could be used outside of the contracts module. For instance, we could use this use mechanism for running quick experiments with on-chain data, quickly testing new versions of runtimes at the development time or before an upgrade.