This summarizes the requirements for a tool that could be used to conveniently run the state test files and possibly be used by fuzzing infrastructure as mentioned by @holiman #4613 (comment)

It looks like testeth currently is not too far from these requirements.

Summary of requirements

Input

• --statetest <path>
• --json - boolean flag, output json output for each opcode or not (it's useful to disable json when benchmarking)
• --nomemory - disable showing the full memory output for each op

Output

{"pc":0,"op":96,"gas":"0x2540be400","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"depth":1,"error":null,"opName":"PUSH1"}
{"pc":2,"op":0,"gas":"0x2540be3fd","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0x40"],"depth":1,"error":null,"opName":"STOP"}
{"output":"","gasUsed":"0x3","time":141485}

It should output a json object for each operation. Example:

{"pc":0,"op":96,"gas":"0x2540be400","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"depth":1,"error":null,"opName":"PUSH1"}

Required: pc, op, gas, stack, depth
Optional: opName, gasCost, error

The stack, memory and memSize are the values before execution of the op.

At the end of execution, some summarical info is good, e.g.

{"output":"","gasUsed":"0x3","time":141485}

Another thing mentioned by Martin #4613 (comment) :

One important aspect is that the execution should not hold on to the data during the entire execution, and spit it out as a json array. Instead, it should output jsonl, one line per op.

testeth status

1. --testfile parameter -- should work after fix testeth -t <TestSuite> -- --singletest <path> <testname> running a test from file #5750 is finished. However -t GeneralStateTests probably will still be required, for now it's easier for us to leave it.

So the command line to run a test would look like

test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {}

2. Disabling json output for each opcode. Omitting --jsontrace disables it.

test/testeth -t GeneralStateTests -- --singletest /home/andrei/dev/sar00.json sar00

However, it will not output any summary (output, gasUsed, stateRoot etc.) either in this case.
We need to fix this (always output json summary in --testfile mode)

3. Disabling memory output. This works with --jsontrace '{"disableMemory" : true}'

test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {"disableMemory" : true}

(there're also options to disable stack and storage output)

4. Output currently looks like this

>test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {}
Test Case "customTestSuite": 
{"depth":"0","gas":"379000","gasCost":"3","memory":[],"op":"PUSH1","pc":"0","stack":[],"storage":{"0x00":"0x03"}}
{"depth":"0","gas":"378997","gasCost":"3","op":"PUSH1","pc":"2","stack":["0x00"]}
{"depth":"0","gas":"378994","gasCost":"3","op":"SAR","pc":"4","stack":["0x00","0x00"]}
{"depth":"0","gas":"378991","gasCost":"3","op":"PUSH1","pc":"5","stack":["0x00"]}
{"depth":"0","gas":"378988","gasCost":"0","op":"SSTORE","pc":"7","stack":["0x00","0x00"]}
{"depth":"0","gas":"373988","gasCost":"0","op":"STOP","pc":"8","stack":[],"storage":{}}
{"stateRoot": "a4b37b548bc12ee198ce885d83b511302d9d8f6b2ec5775618b3893c60e0dc75"}
*** No errors detected

This looks close, but

a. There are additional lines on top and at the bottom. They come from boost.test, but we could look into how to suppress them if that's important.
b. We need to add memSize field (add this in StandardTrace::operator())
c. op should have opcode number and opName the mnemonic
d. depth is off-by-one comparing to Martin's examples. This also can be adjusted in StandardTrace::operator()
e. We need to add the summary line {"output":"","gasUsed":"0x3","time":141485}
That should be added here:

5. testeth accumulates all the traces of the execution, then in the end outputs all of them (in the same code that I linked above). This can be fixed by modifying StandartTrace class to have an option to output immediately.

@holiman To summarize, there's somewhat quirky CLI to run it, there are some additional non-json lines in the output, the rest will be fixed to match the requirements of evmlab.

cc @winsvega

TODO

• Add memSize field
• Add opName field and change op to have number
• Make depth start with 1
• Add summary line at least with output and gasUsed
• Output summary line in --testfile mode even without --jsontrace
• Output one line per op immediately