I think this is a cool start. Maybe you could model the run report after the test report functionality that generates the report from a template (

planemo/planemo/commands/cmd_test_reports.py

Line 1 in 1b61aa1

"""Module describing the planemo ``test_reports`` command."""

) ?

I don't think we need something human-readable actually - we just want to be able to get the IDs. Wolfgang is parsing planemo -v run atm, which is not really ideal.

(We don't have to use YAML btw, JSON is also fine if you prefer.)

I don't think the report itself needs to be human readable (as opposed to what the docs say), I think what's cool is that we can start from the same JSON and produce different kind of reports (markdown, HTML, xunit, plain text, yaml).

I was just hacking on that invocation test report for https://github.com/galaxyproject/planemo/pull/1152/files#diff-27ed15e8e508ef68adea5c98c2025fc3e70937c5d14affdaeaa59b56ba25a6dcR386. It would be nice if this exact JSON is something we could put in the test result JSON somewhere - maybe as a subset of the "data" key or a an "invocation_details" key below "data".

I guess that is just an implementation detail though.

@jmchilton do you mean the json structure should look different so it can be reused?

Yeah - let me be more concrete. I think it should be called execution_report instead of invocation_report since run could be done on tools also and in non-Galaxy contexts.

Move:

        invocation_details = {
            'invocation_id': run_result._invocation_id,
            'history_id': run_result._history_id,
            'workflow_id': run_result._workflow_id,
            'invocation_state': run_result.invocation_state,
            'history_state': run_result.history_state,
            'error_message': run_result.error_message,
        }

Into an execution_details() method on the GalaxyWorkflowRunResponse object and with an empty abstract implementation placed in RunResponse.

Then in def structured_test_data(self, run_response): of planemo/runnable.py also place the details in the data_dict.

data_dict['execution_details'] = run_response.execution_details()

That way tooling that is developed around planemo run for workflows can also be used from the JSON for test cases.

Okay, I'll have a look.

an execution_details() method on the GalaxyWorkflowRunResponse object and with an empty abstract implementation placed in RunResponse.

Aren't we duplicating a lot of stuff here? All this data comes from the GalaxyWorkflowRunResponse in the first place.

There seems to be a step missing, structured_test_data is a method of the TestCase class which is not used when running runnables without testing them.

I can probably get it to work (maybe add a structured_test_data method to the RunResponse class as well?) but otherwise maybe you can clarify what you meant @jmchilton?

I think what I was thinking was this - b8cebeb. But now I see there is a thing called invocation_details that is a lot like what I was proposing with execution_details 😆 - of course @mvdbeek did it first and better.

His general idea of making this into a report option would be doable if we ensure this is all in there in this way (which it may already be though - I'm unsure).

I would be happy to make the simplest change possible:

    invocation_report = kwds.get("invocation_report", None)
    if invocation_report:
        with open(invocation_report, "w") as f:
            json.dump(run_result.invocation_details, f, indent=4, sort_keys=True)

and then modify the collect_invocation_details() method a little bit. But I thought your and @mvdbeek's point was more about making the output compatible with the various report generation options which planemo provides.

But maybe this can be done in a later PR.

I think this might be ready for a review. It produces an output like this: