I'd like to propose something like:

$ kola run-ext-script /path/to/binary

Takes a single binary at /path/to/binary, injects it and a generated basic systemd unit kola-test-ext-$(basename binary).service that runs it.
kola connects via ssh and runs systemctl --wait is-active (does that work? if not find another command).

When the service exits successfully the test is considered successful. When it exits with an EXITCODE error, the test has failed. If it exits from SIGTERM, then we assume the host is rebooting and continue waiting.

Then we support a much bigger generalization of this,

kola run-ext /path/to/dir

which injects a full directory for dependencies, and which would have at least a kola-test.service unit, and maybe we support a container image?

This would allow us to more easily adapt e.g. the ostree test suite to run inside kola. So in ostree's upstream CI we could use that. And importantly the reverse works - run the ostree tests downstream too, i.e. reverse dependency testing. e.g. in FCOS pipeline, when the kernel/systemd change, run ostree's tests.