Scope - third-party tools integration

The idea is relatively simple: some mutations don't make sense, according to other tools that are completely third-party to infection.

Such tools could say things like:

• == cannot exist, according to our coding rules

• we never negate conditionals, because it is confusing to us

• we have strict types

• all methods must be public (silly - please don't do this, but just here for the sake of the example)

Hacks done so far

I've tinkered a lot on https://github.com/Roave/infection-static-analysis-plugin last week.

The tool currently operates through AOP by replacing the Infection\Mutant\MutantExecutionResultFactory service. Whenever a mutant result is produced, we analyze it, and decide whether it is valid as an escaped mutant through a third-party tool.

If the third-party tool says that the mutation is invalid, it is marked as KILLED.

Wished API

Plugins can be either very lightweight (#697, for example, is about adding a regex to a mutation) or very heavyweight (running complex static analysis, fuzzy-testing or property tests on top of a mutation).

Therefore, it makes sense to allow both pre- and post- filtering of mutations.

Pre-filtering

The ideal location where pre-filtering could be done is inside Infection\Process\Runner\MutationTestingRunner#run():

• infection/src/Process/Runner/MutationTestingRunner.php

  Lines 90 to 124 in 25582f2

  	$processes = take($mutations)
  	->map(function (Mutation $mutation): Mutant {
  	return $this->mutantFactory->create($mutation);
  	})
  	->filter(function (Mutant $mutant) {
  	// It's a proxy call to Mutation, can be done one stage up
  	if ($mutant->isCoveredByTest()) {
  	return true;
  	}
  	$this->eventDispatcher->dispatch(new MutantProcessWasFinished(
  	MutantExecutionResult::createFromNonCoveredMutant($mutant)
  	));
  	return false;
  	})
  	->filter(function (Mutant $mutant) {
  	if ($mutant->getMutation()->getNominalTestExecutionTime() < $this->timeout) {
  	return true;
  	}
  	$this->eventDispatcher->dispatch(new MutantProcessWasFinished(
  	MutantExecutionResult::createFromTimeSkippedMutant($mutant)
  	));
  	return false;
  	})
  	->map(function (Mutant $mutant) use ($testFrameworkExtraOptions): ProcessBearer {
  	$this->fileSystem->dumpFile($mutant->getFilePath(), $mutant->getMutatedCode());
  	$process = $this->processFactory->createProcessForMutant($mutant, $testFrameworkExtraOptions);
  	return $process;
  	})
  	;

Here, we could have a replaceable filter for mutations with a signature like:

@psalm-type PreFilterMutant callable(Mutant $mutant): bool {}

or

preFilterMutant :: Mutant -> IO Bool

(reason for IO here is that external tools are gonna be shitty anyway: embrace the shittyness)

This is where static analysis could be plugged in, or #697 could be solved: invalid code mutations can be filtered with a very quick operation, saving a ton of CPU before runtime.

Note: we could also ->map() here, to allow replacing a Mutant

Post-filtering

Post-filtering is more tricky, because of how mutations are considered "done", and how mutation execution progress is tracked.

This happens through relatively ugly callbacks in Infection\Process\Runner\MutationTestingRunner#run():

In Infection\Process\Factory\MutantProcessFactory#createProcessForMutant() this becomes even more muddy:

This is very un-manageable: I think it would make sense to use something like an Observable (/cc @WyriHaximus if you can help out with a suggestion here) to which we publish the completion of the process, rather than publishing to an event dispatcher.

We can then filter said observable exactly like we did with the pre-filtering above:

@psalm-type PostFilterMutant callable(MutantProcess $mutant): bool {}

or

postFilterMutant :: MutantProcess -> IO Bool

Then:

$completedMutationRuns->filter(function (MutantProcess $process): bool {
    return 1 === \preg_match('/potato/', $process->getCommandLine()); // don't show potatoes in the result
});

Note: we could also ->map() here, to allow replacing a MutantProcess

@internal and backward compatibility promise

Few things in this issue require some added maintenance effort for @infection maintainers:

• Some of the Container API needs to become public: specifically for setting a pre- and post- filter callback:
  

  infection/src/Container.php

  Lines 144 to 147 in 25582f2

  	/**
  	* @internal
  	*/
  	final class Container

• Mutant needs to become public API to some degree
  

  infection/src/Mutant/Mutant.php

  Lines 41 to 45 in 25582f2

  	/**
  	* @internal
  	* @final
  	*/
  	class Mutant

• Some of the MutantProcess API needs to become public
  

  infection/src/Process/MutantProcess.php

  Lines 43 to 47 in 25582f2

  	/**
  	* @internal
  	* @final
  	*/
  	class MutantProcess implements ProcessBearer

That means that before working on this, there needs to be a clear idea by maintainers of the project whether those symbols can be made public, or whether more work needs to be done around them first (https://twitter.com/tfidry/status/1302896716065144833 /cc @theofidry )