Please always use override for virtual functions, and perhaps even virtual in the derived class.

Please mention that this is the common superclass of syntax and semantics tests

Also please document the virtual functions.

Please use using x = y instead of typedef.

This now returns a bool, since "no return data" will be considered the same as "revert" - so far I haven't found a reliable way to distinguish between the two using the cpp-ethereum JSON RPC interface.

How about:

f(bool,uint)->(uint,uint)
true|2->2|3

To explain: first parentheses contains the argument types, comma separated, while the second one contains the return parameters, comma separated.

The expectations follow the same pattern, but instead of comma are separated by the pipe symbol, in case the comma could be a valid token in inputs. But I think comma should be fine, since we only support dot for number literals.

Ok, now I get it. I think adding parentheses in the second line might help reading this:

f(bool,uint)->(uint,uint)
(true|2)->(2|3)

And maybe using commas instead of pipes in the second line might be fine as well (I'm not sure commas will need to be a valid input token outside of strings and within strings pipes may occur as well...).

This syntax has the advantage of making the desired encoding of the return values explicit (at the moment I'm trying to deduce the types from an untyped-list and try to re-encode the actual results using those deduced types, but that's probably more error-prone).

However, using typed return values will make it more difficult to support supplying unaligned/weird inputs - although we might just use special return types for that (like "unpaddedbytes" that can be encoded e.g. as a hex string).

@chriseth What do you think?

How about f(bool,uint)->raw for the special case?

e.g.

f(bool,uint)->raw
(true,false)->0011442200440..00

That's what I had in mind, yes - but now I realize: that still won't allow you to provide malformed, resp. strangely encoded inputs... and we can't change the left hand side of the first line to f(raw), since we need the correct signature for the call.

That's related to @chriseth's comment in the issue, i.e. the idea was not force you to use correctly ABI-encoded results as well as inputs in order to be able to check the behavior if a function is supplied with malformed input.

That's why currently the input encoding is completely independent of the function signature and no return signature is provided at all.

We could allow something like:

f(bool,uint)->raw
raw:112233445566...FF->00114422004400..00

But I'm not sure that's better than the current solution of generally using an untyped encoding both for input and return values.

How about this then (also whitespaces should be allowed below):

f(bool,uint)->(uint,uint)
true,1->1,2
f(bool,uint)->(uint,uint)
raw:00aa112200->raw:00110044

If the section starts with raw: then it is considered raw, independent from the types declared.

This will make it difficult for isoltest to realize that it should re-encode mismatching actual results as raw bytes again and not to use two uint's instead, so I think

f(bool,uint)->raw
raw:001122...->raw:001122

would probably be better. The actual return signature of the function is not parsed anyways, so specifying it in the test expectations won't have an effect apart from guiding parsing and re-encoding of return values.

Furthermore, actually using the signatures plus the raw construction won't allow it to mix regularly encoded and raw bytes, but that's probably not needed anyways.

In general the first thing we need to agree on is whether we want to use the function signature at all for encoding the input - if we agree on that, I could already start changing the PR accordingly.

As a reminder: so far this PR ignores the function signature entirely and input arguments are specified by a comma separated list of values whose type is automatically deduced and padded to 32-bytes by default, but padding can be disabled using unpadded, so:

f(bool,uint): true, 1234
-> 0x11, 0x22

currently will encode the input as two 32-bytes blocks and expect two 32-bytes blocks as output, and

f(bool,uint): unpadded(hex"00000000000000000000000000000000", 1), 1234
-> 0x11, 0x22

will be the same (the hex string is 31 bytes and the single 1 will be the 32nd byte, which together is the standard 32-bytes encoding of the bool true).

So shall we discard this and instead use the function signature to guide the encoding by default? @chriseth What do you think?

@axic Another issue to think about is struct's - we could support them in the signature as well and have nested parentheses when specifying arguments, but it will complicate things further and it will be easier to deal with them using the current "untyped" solution.

Probably the ability to specify incorrectly encoded input and output should not have a high priority.
Still, the current solutions are not exactly what I would call readable. Why do we have to specify the types at all? If we go down that route, we might consider an earlier suggestion and use

f(true, 7) == 7, 8

where isoltest just asks the compiler for the types. The case of multiple functions with the same name might also be an edge case that can be solved with raw encoding.

Structs could be out of scope for now.

Hm. The current solution is the so far closest to the current tests in SolidityEndToEndTest.cpp.

The main question is probably whether we want to let the encoding of arguments and return values actually depend on the function (and return) signature at all or not. That would mean to use some kind of ABI encode/decode and the tests would depend on those to be correct. If one goes down that road, I wonder why one wouldn't just move the check into the contract instead, i.e.:

contract C {
    function f(bool a, int b) returns (uint, uint) { ... }
    function test() returns (bool) { return f(true, 7) == (7, 8); }
}

And then just verify whether test returns true (well, something like that - there's currently no tuple comparisons and one would need to care about internal and external calls, etc., but I think you get the idea).

In case anybody wonders: this is only in here as part of #4305, on which this PR depends and which is not yet merged.