wasm2c is failing several spec tests if we add -O or -O2 to the compiler command line.

• Example commit here: fix-project@551b3db
• Corresponding CI output on Ubuntu: https://github.com/fixpointOS/wabt/runs/6547538885?check_suite_focus=true

There seem to be at least a few buckets:

1. The WebAssembly spec requires that "There is no observable difference between quiet and signalling NaNs," and the spec tests enforce that some operations transform signalling NaNs into quiet NaNs (e.g. here: https://github.com/WebAssembly/spec/blob/main/test/core/float_exprs.wast#L2361).

   But, gcc and clang often make optimizations that preserve signalling NaNs, even when the same code, compiled without optimization, will turn a signalling NaN into a quiet NaN. E.g. in this code which is essentially what wasm2c generates for a promote/demote:

   float func(float x)
   {
     double y = x;
     return y;
   }

   On x86-64 and compiling without optimization, gcc and clang will turn this into a cvtss2sd/cvtsd2ss, which turns a signalling NaN into a quiet NaN (passing the tests). If we compiled with -O, the function turns into a NOP and the signalling NaN is preserved (breaking the tests). I couldn't find a way to change this behavior with fenv(3) or with a -fno-... optimization flag. One possible solution might be to mark all floating-point stack variables as volatile, but this may be a heavy price in terms of slowdown. Maybe there's a path forward that involves explicitly clearing a signalling NaN only when (a) storing into linear memory, (b) reinterpreting as integer, or (c) returning to a host function? It might be hard to guarantee this catches all the edge cases...

2. The spec requires OOB loads/stores to trap deterministically, but when using mprotect & a signal handler to detect OOB memory access, gcc/clang -O omits loads/stores if they don't affect the output or a visible side-effect (which the segfault doesn't count as). E.g. even this test fails: https://github.com/WebAssembly/wabt/blob/main/test/wasm2c/address-overflow.txt

   I'm curious how this is done in other Wasm engines that also use the "signal handler trick" to detect OOB memory access (while, hopefully, also optimizing to remove dead code while still passing all the spec tests)?

3. The spec tests say, "Implementations are required to have every call consume some abstract resource towards exhausting some abstract finite limit, such that infinitely recursive test cases reliably trap in finite time," and enforce this, e.g. here: https://github.com/WebAssembly/spec/blob/master/test/core/call.wast#L337 But, I think gcc/clang -O are optimizing these kinds of runaway recursive functions into infinite iteration/proper tail recursion, so the exhaustion segfault never arrives. (I fear this issue was recently introduced by wasm2c: use signal handler to detect stack exhaustion #1875.)

For our purposes, we'd like to compile wasm2c output with optimization, and we also want the "almost-deterministic"/spec-conforming behavior of WebAssembly, so it seems worthwhile to get the tests passing even with -O. I'd love to hear if this has been discussed before and what others think about the best route (@binji @sbc100 @kripken ?).