fix(assets): support exceptions to exclude patterns#4473
fix(assets): support exceptions to exclude patterns#4473mergify[bot] merged 44 commits intoaws:masterfrom
Conversation
* additional test cases * refactor existing tests
|
Thanks so much for taking the time to contribute to the AWS CDK ❤️ We will shortly assign someone to review this pull request and help get it
|
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
| if (stat.isDirectory()) { | ||
| // to help future shouldExclude calls, we're changing the exlusion patterns | ||
| // by expliciting "dir" exclusions to "dir/*" (same with "!dir" -> "!dir/*") | ||
| exclude = exclude.reduce<string[]>((res, pattern) => { |
There was a problem hiding this comment.
I don't think this should be done in-line while traversing. It is also possible to run through the list of entries in the ignore list, see which ones are directories and treat those as prefixes.
There was a problem hiding this comment.
In fact, I'd like to convert the "list of exclude/include entries" into a function or class that encapsulates the behavior as soon as possible and pass that around.
I'm thinking something like:
const ignoreList = parseIgnoreList(rootDirectory, ...list of strings);
ignoreList will be either of type (path: string) => boolean, or be a class with a method of that type.
I would have that as an argument to listFilesRecursively().
There was a problem hiding this comment.
The issue is that it wouldn't be very economical, fs calls-wise. I'm filling the exclude directory on the fly because I'm also checking the stat of the files I'm traversing.
| @@ -0,0 +1,22 @@ | |||
| # This a comment, followed by an empty line | |||
There was a problem hiding this comment.
Can we do these tests in memory?
I'm thinking of a test where we have preloaded this list of ignore patterns, and pushing a number of paths past it (either by means of shouldExclude() or by means of a new IgnoreList API)
That is effectively the same and easier to change/assert against than making a whole new fixture with "just these" files in it.
There was a problem hiding this comment.
I ended up creating a createFsStructureFromTree function, and removing the previous fixtures (with the exception of demo-image/).
Does that work for you?
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
| @@ -0,0 +1,98 @@ | |||
| import fs = require('fs'); | |||
There was a problem hiding this comment.
I'm not sure whether it's necessary to upgrade this to a feature of the assert library yet, especially since it doesn't really assert anything, will not be used by most of the construct libraries, and will never be JSII-able.
At least we can just keep this as a testing utility in the assets library until we definitely need it somewhere else, no?
There was a problem hiding this comment.
I moved it to assert because I was using it both in assets and aws-ecr-assets. I could move it to assets instead, and expose it to make to available to aws-ecr-assets?
There was a problem hiding this comment.
I see. Yes, I think put it in assets, and in fact you could import it directly from the test/ directory in aws-ecr-assets:
import fsUtils = require('@aws-cdk/assets/test/fs-utils');It's not super nice, but I don't want test helpers to be part of the public API of a package. This usage would be okay by me since it's between 1st-party packages only. Would be nice to spend a couple of comment lines on it when you're making that change.
| * @returns `true` if the file should be excluded, followed by the index of the rule applied | ||
| */ | ||
| export function shouldExcludePriority(exclude: string[], filePath: string): [boolean, number] { | ||
| return exclude.reduce<[boolean, number]>((res, pattern, patternIndex) => { |
There was a problem hiding this comment.
I know reduce is sexy to use, but since you're not even using current accumulation res in your reducer, I don't really see the value. What's wrong with a simple for loop that returns the last set value?
There was a problem hiding this comment.
I was originally using a map to have both pattern and patternIndex. I'll revert back to it.
| * | ||
| * @returns `true` if the file should be excluded, followed by the index of the rule applied | ||
| */ | ||
| export function shouldExcludePriority(exclude: string[], filePath: string): [boolean, number] { |
There was a problem hiding this comment.
Does this function really need to be exported?
There was a problem hiding this comment.
The name confuses me. Proposed name change: how about evaluateExcludePatterns.
Or in fact, I've been saying this a couple of times already: please put these related functions that operate on the same data structure in a class. They're not as independent as you would like.
class ExcludeRules {
constructor(private readonly patterns: string[]) { }
public excludeFile(filePath: string): boolean { ... }
public excludeDirectory(dirPath: string): boolean { ... }
private evaluate(pathName: string): [boolean, number] { ... }
}There was a problem hiding this comment.
I've switched to a class, that I'm still exporting to be able to test it. Same for your evaluate/my evaluateFile, it needs to be public.
| * @param exclude exclusion patterns | ||
| * @param filePath file path to be assessed against the pattern | ||
| */ | ||
| export function shouldExcludeDeep(exclude: string[], relativePath: string): boolean { |
There was a problem hiding this comment.
Does this function really need to be exported?
There was a problem hiding this comment.
I was exporting it to test it. It's the same now with the public method
| * @param exclude exclusion patterns | ||
| * @param directoryPath directory path to be assessed against the pattern | ||
| */ | ||
| export function shouldExcludeDirectory(exclude: string[], directoryPath: string): boolean { |
There was a problem hiding this comment.
Does this function really need to be exported?
There was a problem hiding this comment.
I was exporting it to test it. It's the same now with the public method
| export function shouldExcludeDeep(exclude: string[], relativePath: string): boolean { | ||
| const [_shouldExclude] = relativePath.split(path.sep).reduce<[boolean, number, string]>( | ||
| ([accExclude, accPriority, pathIterator], pathComponent) => { | ||
| pathIterator = path.join(pathIterator, pathComponent); |
There was a problem hiding this comment.
I feel it would be cleaner if you made a function like pathComponents() (which given a path like a/b/c would return [a, a/b, a/b/c]) and iterated over that.
Again I feel a simple for loop would be more natural. If you really want to make it functional, why not something like:
const resultsAndPrios = pathComponents(relativePath).map(pc => shouldExcludePriority(rules, pc));
const [shouldExclude] = pickHighest(resultsAndPrios, x => x[1]);
return shouldExclude;Requires you to write some helpers, but to me is a lot clearer what is happening.
There was a problem hiding this comment.
I did some refactoring, preferring for loops over reduces. I've only kept maps when I needed to know both the current array value and index.
|
|
||
| /** | ||
| * Determines whether a given directory should be excluded and not explored further | ||
| * This might be true even if the directory is explicitly excluded, |
There was a problem hiding this comment.
In the case you're describing, shouldn't the function return false rather than `true?
| test.done(); | ||
| }, | ||
| 'contridactory'(test: Test) { | ||
| testShouldExcludeDeep(test, ['foo.txt', '!foo.txt'], [], ['foo.txt']); |
There was a problem hiding this comment.
How about contradictory with the order reversed?
| } | ||
| }, | ||
|
|
||
| shouldExcludeDeep: { |
There was a problem hiding this comment.
Thanks for a thorough test suite!
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
|
Thank you for contributing! Your pull request is now being automatically merged. |
|
Thanks again @rix0rrr 🎉 |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
* chore: better test * fix: folder exclusion * additional test cases * refactor existing tests * chore: add new test files * fix: listFilesRecursively refactor (wip) * fix: finish refactoring listFilesRecursively * fix: implement mkdirpSync * fix: symlink discovery * fix: don't follow symlinks early * fix: create empty directories * chore: remove useless let * fix: symlink fingerprinting * fix: don't throw when fingerprinting directories * chore: remove unneeded 'exclude' cloning * chore: refactor mkdirp calls * chore: refactor AssetFile * chore: refactor recursion * chore: prevent unneeded stats call * chore: createFsStructureFromTree, remove empty files * chore: cleanup * fix: empty-directory test * feat: shouldExcludeDeep * chore: fromTree in @/assert, cleanup fn, test * feat: shouldExcludeDirectory * chore: refactor listFiles with new methods (missing symlinks) * feat: add symlink support to fromTree * fix: fromTree external directory symlink * fix: listFiles symlink support * chore: additional contridactory test * chore: fix docs * chore: ExcludeRules class refactor * chore: evaluateFile refactor * chore: further evaluateFile refactor * chore: evaluateDirectory refactor * chore: move FsUtils to assets * chore: move FsUtils to assets (unstaged files)
…nto 4295-windows-ecs-support * '4295-windows-ecs-support' of github.com:arhea/aws-cdk: chore(deps-dev): bump @types/lodash from 4.14.146 to 4.14.147 (aws#5021) Revert "fix(assets): support exceptions to exclude patterns (aws#4473)" (aws#5022) chore(deps): bump jsii-pacmak from 0.20.3 to 0.20.5 (aws#5003) chore(deps): bump codemaker from 0.20.3 to 0.20.5 (aws#5007) chore(deps-dev): bump @types/jest from 24.0.22 to 24.0.23 (aws#4993) chore(deps): bump jsii from 0.20.3 to 0.20.5 (aws#5006) chore(deps-dev): bump jsii-diff from 0.20.3 to 0.20.5 (aws#5005) chore(deps): bump jsii-spec from 0.20.3 to 0.20.5 (aws#5008) chore(core): resolve tokens before publishing tree.json (aws#4984) feat(cli): adding new option to `cdk deploy` to indicate whether ChangeSet should be executed (aws#4852) chore: move semantic.yaml to .github/ fix(core): unable to find stack by name using the cli in legacy mode (aws#4998) fix(ecs-patterns): Fix issue related to protocol being passed to target group (aws#4988)
Allows adding back files and directories from a previously excluded directory. Previous implementations of
copyDirectoryandfingerprintstopped walking through the directory if it was ignored (see comment in #4450)direxclusions intodir/*, to allowminimistto match files in folderlistFilesRecursivelymethod, removing duped code fromcopyandfingerprint.dockerignoretest cases into@aws-cdk/aws-ecr-assetsFixes #4450
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license