we grep for the debug output and because of the leading ! of this line is the exit code of the last pipe inverted.

means we are getting only a successfull exit code, when the text was not matched

This grep business can break really easily.

What I'd like instead:

1. Add isResultCacheUsed to AnalysisResult (pass $resultCache->isFullAnalysis() to the constructor)
2. Create FailWithoutResultCacheErrorFormatter. After it checks that result cache was used, we can just call TableErrorFormatter.
3. Use this error formatter here in this test instead of grep.

added another test case, because I realized phpstan is also not using the result cache the other way arround

This grep business can break really easily.

What I'd like instead:

1. Add isResultCacheUsed to AnalysisResult (pass $resultCache->isFullAnalysis() to the constructor)
2. Create FailWithoutResultCacheErrorFormatter. After it checks that result cache was used, we can just call TableErrorFormatter.
3. Use this error formatter here in this test instead of grep.

I just checked AnalyseCommand and the codepath with --generate-baseline actually isn't influenced by the selected error formatter. So this isn't going to fail.

I really like --error-format=failWithoutResultCache and am looking forward to use it in other E2E tests, but not sure what to do about it here.

I had no better idea either. I went back using the grep way for the --generate-baseline case.

to make it more robust, I am doing a initial run, which does not expect the error message beeing in stdout and the subsequent run with the very same string.

hopefully thats good enough.

Sleeping over it I got a different idea.

What about adding a new cli option, e.g --assert-result-cached or --fail-when-no-result-cache instead of using a error-formatter?

edit: maybe instead we could trigger it via ENV var, so the enduser won't see this testing-only infrastructure

implemented a ENV var approach in phpstan/phpstan-src#2611