Interesting idea. I remember @sunfishcode was saying we should use SimpleJIT for testing the runtime behavior and correctness of Cranelift in the past; we just never got to it. I think doing it directly with clif-util makes sense. I've made a list of things that came to mind while reading this code, and could be addressed either in this PR or in subsequent ones.

I'd be worried that testing only specific signatures could mean a lot of missing test coverage for calls to functions with other signatures; but then having to deal with the Rust ABI is a bit unfortunate. Having functions that return only booleans, means that we can use a helper function to test that a call works, but we stay in Cranelift jit territory (and if there are consistent errors within Cranelift, they couldn't be caught). Which might be good enough for a start.

• I wonder if we could return more complicated types, like i64 or a simd 128-bit vector type, that would be used in assertions in comments; we'd have comments like assert_returns i64 42.

This would be good to have in automation too, but we'd need to limit running the runtime tests to automation host architectures that are supported by Cranelift. And by not having much control over Travis CI, not sure we can run on e.g. ARM64 in the future. Worth nothing at least.

• Use this in automation.

Also to be entirely useful, I wonder if this shouldn't run the function in another process, in case the called function causes a segfault / other kind of hard crash. Otherwise, this means the first function that crashes causes the rest of the functions to not run, which is unfortunate.

• Handle segfaults one way or another.

I wonder if this shouldn't run the function in another process, in case the called function causes a segfault / other kind of hard crash.

fork() should work on *nix, though it should be called as close to the call of the codegened function as possible. Also after returning from the codegened function, you should immediately call std::process::exit(0). This is to avoid weird stuff. fork makes the new process single threaded for example, and also forbids calling certain libc functions (calling those is UB).

For your information, I have been discussing this approach with @abrown over IRC, and suggested to use a single and only signature for function calls as a feature.

If you want to check the generated code for making a function call with a given ABI, this can already be done without the need for executing the code, with ordinary tests using ; check: and ; nextln: and even ; bin:.

So executing the code is just a matter of being able to call a single function. What I initially suggested was to have a ; run: %foo(1, 2, 3) == 51 which would generate a tiny function, compiled with Cranelift, with the signature () -> bool and which calls %foo with the expected argument and ABI.

I think the approach taken here goes in the right direction. So, even if it feels incomplete on many aspects, we should probably not block this work with too many wishes.

• About checking complex type content, this is something which we can add vocabulary for as we improve this system. With this patch we are able to call a () -> bool function. Generating such a function for calling other signature and asserting other properties sounds like incremental future work.
• About running other architecture, this is something we have done for SpiderMonkey testing already, by importing the simulators of other architectures (arm32, arm64, mips*). Another approach would be to rely on qemu and generate binaries.
• About running in different processes, I think this would be premature at the moment, but a nice to have feature that I would probably not implement my-self until it pains me enough. Thus we should not block this feature on it.

Yeah yeah, I wasn't suggesting to mandate all of this before merging. More thinking about the running in different processes part had me realize that it'd be nice to have this, in two modes: 1. for running the tests in CI where each test is run in its own process (to not crash the test process itself), 2. for debugging tests, we would probably prefer something as is right now, that is, a test (or collection of tests) running in the test process itself, so it's easier to run within gdb/rr etc.

I'm all for fork-ing to avoid crashes, the run: %foo(...) == ... syntax @nbp is talking about, and wiring this up to the automation so that we always know that some CLIF actually runs (especially the x86 SIMD stuff 😄)--I do think all of those could be future PRs, though.

I am interested in using SimpleJIT here but someone will have to help me out. I looked at it for a moment but how to use it was obscured by the module-level stuff--I really just needed Context::compile and Context::emit_to_memory (https://github.com/CraneStation/cranelift/pull/872/files#diff-8643ebb587e63cc619cd28e00be6a7e5R127-R131).

The more I think about it, the more unclear it is that we can actually use SimpleJIT for this, especially considering how simple the code in the run pass is, so this looks fine as is to me in the current state. Thanks for doing this!

@sunfishcode rebased this on current master; should be ready to review.

@bnjbvr, I've resolved the issues above and pushed to https://github.com/abrown/cranelift/tree/clif-filetests-test-run so that we can merge as a part of #890.