Skip to content

Cache on successful file processing#3614

Merged
TomasVotruba merged 2 commits intorectorphp:mainfrom
yguedidi:cache-on-successful-file-processing
May 1, 2023
Merged

Cache on successful file processing#3614
TomasVotruba merged 2 commits intorectorphp:mainfrom
yguedidi:cache-on-successful-file-processing

Conversation

@yguedidi
Copy link
Copy Markdown
Contributor

@yguedidi yguedidi commented Apr 13, 2023

New version of #3604
Closes rectorphp/rector#7770
Now a file will be cached only when not in dry run mode

needs first:

@yguedidi yguedidi requested a review from TomasVotruba as a code owner April 13, 2023 22:58

if ($errorAndFileDiffs[Bridge::SYSTEM_ERRORS] !== []) {
$this->changedFilesDetector->invalidateFile($file->getFilePath());
} elseif (! $configuration->isDryRun()) {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the added condition


if ($systemErrorsAndFileDiffs[Bridge::SYSTEM_ERRORS] !== []) {
$this->changedFilesDetector->invalidateFile($file->getFilePath());
} elseif (! $configuration->isDryRun()) {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the added condition

@samsonasik
Copy link
Copy Markdown
Member

I think this one need e2e test to avoid regression and back and forth check, eg:

  • multiple --dry-run check should ok.
  • multiple succesfull keep changing, eg, the following code got twice changed should ok.
if ($a && $b) {
    return true;
}

on config:

<?php

declare(strict_types=1);

use Rector\CodingStyle\Rector\Stmt\NewlineAfterStatementRector;
use Rector\Config\RectorConfig;
use Rector\EarlyReturn\Rector\If_\ChangeAndIfToEarlyReturnRector;

return static function (RectorConfig $rectorConfig): void {
    $rectorConfig->rule(ChangeAndIfToEarlyReturnRector::class);
    $rectorConfig->rule(NewlineAfterStatementRector::class);
};

First: https://getrector.com/demo/55b308a8-af17-48cb-b2be-9c0d26ab60c4
Second: https://getrector.com/demo/53cd71fb-1851-44d0-aa1e-ba7e5329db77

@yguedidi
Copy link
Copy Markdown
Contributor Author

Thanks for guidance! Will work on separate PRs this week-end to add those e2e tests

@yguedidi yguedidi marked this pull request as draft April 14, 2023 22:17
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch 2 times, most recently from f8f8588 to ceac622 Compare April 15, 2023 00:04
@staabm
Copy link
Copy Markdown
Contributor

staabm commented Apr 22, 2023

Is this still WIP ?

@samsonasik
Copy link
Copy Markdown
Member

It needs one more e2e test for consecutive succesfull.

for consecutive dry-run, there was already PR #3616

@yguedidi
Copy link
Copy Markdown
Contributor Author

I planned to work on that e2e test this week-end

@yguedidi
Copy link
Copy Markdown
Contributor Author

@samsonasik @staabm here it is: #3666

@@ -32,7 +32,7 @@ public function testHasFileChanged(): void
$filePath = __DIR__ . '/Source/file.php';
Copy link
Copy Markdown
Contributor

@staabm staabm Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these test-changes required?

existing tests shouldn't be changed with new PRs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because previous logic is splitted in 2 methods now, I renamed one so maybe it's clearer

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 092b701 to 58e34f2 Compare April 28, 2023 09:14
@staabm
Copy link
Copy Markdown
Contributor

staabm commented Apr 28, 2023

I think the PR itself needs a new test-case reproducing the issue it is trying to fix

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch 2 times, most recently from c323a5e to a6d89fc Compare April 28, 2023 09:43
@yguedidi
Copy link
Copy Markdown
Contributor Author

@staabm good point! I'll try to come with a test in following days, but the final code is ready if you want to have a look, see that all tests pass now, even the consecutive runs test.
@samsonasik Idea is to cache files only when Rector didn't changed them, so when they are in their final state. So following runs will still go through previously changed files to attempt to change them again.
Maybe not the best way, but at least it preserve actual Rector vision on multiple runs and makes the cache reliable

Credits to @stof for the idea! Thank you

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from a6d89fc to 7635179 Compare May 1, 2023 11:44
@samsonasik
Copy link
Copy Markdown
Member

Please rename commit message with double quote "" , that will cause failure build downgrade

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 7635179 to 2e10efa Compare May 1, 2023 12:01
@yguedidi
Copy link
Copy Markdown
Contributor Author

yguedidi commented May 1, 2023

@samsonasik that revert commit is a temporary one just to generate a failing build, to show the issue.
will remove it from the final PR, showing that the PR solves the issue

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 2e10efa to 489b03b Compare May 1, 2023 12:04
@yguedidi
Copy link
Copy Markdown
Contributor Author

yguedidi commented May 1, 2023

Here the failing test without the fix: https://github.com/rectorphp/rector-src/actions/runs/4850942011/jobs/8644319975?pr=3614
See that with the fix by caching at the end instead of caching at the beginning, that previously failing test is now green: https://github.com/rectorphp/rector-src/actions/runs/4850961546/jobs/8644357227?pr=3614

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we expect the timeout error to stay even with cache, and now the timouted files are not cached because we cache on success instead of caching at the beginning of the processing


return static function (RectorConfig $rectorConfig): void {
$rectorConfig->cacheClass(FileCacheStorage::class);
$rectorConfig->parallel(1);
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used 1 second timeout to maximize chances of timeout

__DIR__ . '/src',
]);

$rectorConfig->sets([LevelSetList::UP_TO_PHP_82]);
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used a big rule set to maximize chances of timeout

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used a big file from src to maximize chances of timeout

@yguedidi yguedidi marked this pull request as ready for review May 1, 2023 12:11
@yguedidi
Copy link
Copy Markdown
Contributor Author

yguedidi commented May 1, 2023

@TomasVotruba @samsonasik @staabm this is now ready for review!

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 489b03b to b7bc6c4 Compare May 1, 2023 12:17
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from b7bc6c4 to 53cfe73 Compare May 1, 2023 12:30
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 53cfe73 to cba16d3 Compare May 1, 2023 12:40
@yguedidi yguedidi requested a review from samsonasik May 1, 2023 12:41
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from cba16d3 to cc2811b Compare May 1, 2023 13:05
Copy link
Copy Markdown
Member

@samsonasik samsonasik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good for me 👍

@TomasVotruba
Copy link
Copy Markdown
Member

Thank you 👍

@TomasVotruba TomasVotruba merged commit f6972de into rectorphp:main May 1, 2023
@yguedidi yguedidi deleted the cache-on-successful-file-processing branch May 1, 2023 21:55
@staabm
Copy link
Copy Markdown
Contributor

staabm commented May 5, 2023

I think we need the same fix in https://github.com/easy-coding-standard/easy-coding-standard. @yguedidi could you check that?

@yguedidi
Copy link
Copy Markdown
Contributor Author

yguedidi commented May 5, 2023

@staabm I'm not using ECS, so I can't guarantee that I can come with a PR any time soon there sorry.
It's fine if some else inspire from this PR to implement it in ECS :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rector appears to be caching files before they've finished processing.

4 participants