Cache multiple paths and add glob pattern support by ethanis · Pull Request #212 · actions/cache

ethanis · 2020-03-12T00:20:25Z

This adds the ability to specify multiple items to cache in a single run (#16, #180). Additionally, the items to cache can be located with glob patterns. Paths/patterns are specified as new-line delimited input to the path parameter.

Because this adds multiple directories to the tarball that is cached, each path is made relative to the GITHUB_WORKSPACE env variable. If paths are outside of this directory, the tarball can contain a backtraced path (e.g. ../foo/bar). Because of this the -P flag is used with tar when unpacking the cache.

I have a couple of open questions:

If no paths are found (e.g. a glob pattern found no files/directories), should the action fail intentionally?
Because the tarball entries are now relative to GITHUB_WORKSPACE older cache entries will not be compatible with these changes. Is there a preferred way to add a salt value to the cache key?

Some examples:

    - name: Restore multiple relative paths
      uses: actions/cache@v3
      with:
        path: |
          node_modules
          dist
        key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

    - name: Restore multiple absolute paths
      uses: actions/cache@v3
      with:
        path: |
          ~/foo/bar
          /users/mona/cache-me
        key: ${{ runner.os }}-${{ hashFiles('key.txt') }}

    - name: Restore from glob patterns
      uses: actions/cache@v3
      with:
        path: |
          **/*.ts
          **/node_modules
        key: ${{ runner.os }}-${{ hashFiles('key.txt') }}

remove known failing test Quote tar paths Add salt to test cache Try reading input files from manifest bump salt Run test on macos more testing Run caching tests on 3 platforms Run tests on self-hosted Apparently cant reference hosted runners by name Bump salt wait for some time after save more timing out smarter waiting Cache in tmp dir that won't be deleted Use child_process instead of actions/exec Revert tempDir hack bump salt more logging More console logging Use filepath to with cacheHttpClient Test cache restoration Revert temp dir hack debug logging clean up cache.yml testing Bump salt change debug output build actions

dhadka · 2020-03-12T14:05:39Z

Thanks for working on this!

We should look at using the version field in the server API. The idea is you compute a hash of any important settings used by the cache action, which in this case is the path, so when the path changes the version changes. Otherwise, if someone were to say add a new line to the path but forgot to change the key, you would get the old cache entry that does not include the new path.

ethanis · 2020-03-12T16:45:53Z

The version field sounds like a perfect fit - is it exposed as a url param here?

Also, what should the initial value be? 1.0?

ethanis · 2020-03-16T16:22:09Z

src/cacheHttpClient.ts

    const httpClient = createHttpClient();
-    const resource = `cache?keys=${encodeURIComponent(keys.join(","))}`;
+    const version = getCacheVersion();
+    const resource = `cache?version=${version}`;


@dhadka I took a stab at using a version to represent a cache. I'm not entirely sure how the endpoints expose this parameter, so I'm happy to revise as needed 😄

We will still need to provide the keys here. It should be cache?keys=key1,key2,...,keyn&version=....

I also don't think we need to include Key and RestoreKeys in the version. With the version, we only match caches with the exact (key, version) tuple. So having the keys included in the version is redundant.

Got it - updated to not include the keys in the version and include the keys as url params.

@dhadka Why do we not need keys here?

I would think you could have an exact match on version but a restore key partial match since version only takes into account the action version and the input paths

smorimoto · 2020-03-17T04:41:06Z

This is great! Thank you @ethanis!

smorimoto · 2020-03-17T05:07:02Z

Can you change to pass the path as Collections? It will be more yaml-like.

dhadka · 2020-03-17T13:08:25Z

@imbsky Lists are currently not supported, see actions/toolkit#184. Ideally, once that is fixed, we would update this to support collections (as well as remain backwards compatible with multiline strings)

smorimoto · 2020-03-17T14:41:37Z

I see. Well, I don't think there is a big problem so this looks good.

joshmgross

Thanks for this!

As far as I can tell there are no breaking changes, but we may need to push this as a v2 update to the action to be safe.

src/cacheHttpClient.ts

src/tar.ts

ethanis · 2020-03-19T19:51:29Z

@joshmgross is there anything else that needs to be changed in this PR?

boredland · 2020-03-19T21:37:16Z

this finally makes this actions usable without thousands of unncecessary lines when in a yarn monorepo.

dhadka

LGTM from the server side.

joshmgross · 2020-03-20T13:33:48Z

@joshmgross is there anything else that needs to be changed in this PR?

I'll take a look later today, but should be all good

chrispat · 2020-03-20T14:13:21Z

I would like to have one example added as part of this change that uses multiple paths. @boredland perhaps you have one in mind?

boredland · 2020-03-20T14:48:25Z

@chrispat I think the lerna example would be helpful:

- name: restore lerna
     uses: actions/cache@v3
      with:
        path: |
          node_modules
           */*/node_modules
        key: ${{ runner.os }}-${{ hashFiles('yarn.lock') }}

ethanis · 2020-03-20T15:49:34Z

@boredland do lerna projects have multiple yarn.lock files? If so, should the key should be based on all of them?

- name: restore lerna
  uses: actions/cache@v3
   with:
     path: |
        **/node_modules
     key: ${{ runner.os }}-${{ hashFiles('**/yarn.lock') }}

smorimoto · 2020-03-20T16:10:44Z

If I remember correctly, Lerna and yarn workspace have only one lock file at root.

ethanis · 2020-03-20T19:06:55Z

@boredland, @chrispat I added an example for lerna here

joshmgross

Thanks for this contribution!

The current plan is for this to go in v2

examples.md

Co-Authored-By: Josh Gross <joshmgross@github.com>

gcourtemanche · 2020-04-02T10:52:37Z

Thanks for the great work! I'm using Yarn and Lerna, looking forward for v2.

@imbsky Lerna uses yarn.lock at root but also in every package.

@ethanis Is it possible to add in the doc that the v2 is not released yet. I spent some time trying to understand why the example wasn't working.

joshmgross · 2020-04-02T14:08:20Z

Thanks @gcourtemanche! Fixing that here: #242

Bnaya · 2020-04-25T13:54:27Z

Dose is support ignore path somehow ?

      - name: restore yarn workspaces cache
        uses: actions/cache@master
        with:
          path: |
            node_modules
            */*/node_modules
           !packages/notyou/
          key: ${{ runner.os }}-${{ matrix.node }}-${{ hashFiles('**/yarn.lock') }}

joshmgross · 2020-04-27T14:18:25Z

Dose is support ignore path somehow ?

      - name: restore yarn workspaces cache
        uses: actions/cache@master
        with:
          path: |
            node_modules
            */*/node_modules
           !packages/notyou/
          key: ${{ runner.os }}-${{ matrix.node }}-${{ hashFiles('**/yarn.lock') }}

Ignoring paths should be supported. The paths are evaluated by the @actions/glob package, see https://github.com/actions/toolkit/tree/master/packages/glob#exclude-patterns

ethanis and others added 7 commits March 4, 2020 16:02

Allow for multiple line-delimited paths to cache

84cead4

Add initial minimatch support

1e23344

Use @actions/glob for pattern matching

db235cf

unit test coverage for caching multiple dirs

057d9de

Ensure there's a locateable test folder at homedir

26a2f24

Clean up code

ac5351d

ethanis added 2 commits March 16, 2020 09:09

Version cache with all inputs

820f06f

Unit test getCacheVersion

bd756c1

ethanis commented Mar 16, 2020

View reviewed changes

Include keys in getCacheEntry request

072d513

dhadka requested a review from joshmgross March 16, 2020 20:52

joshmgross reviewed Mar 17, 2020

View reviewed changes

src/cacheHttpClient.ts Show resolved Hide resolved

src/tar.ts Show resolved Hide resolved

src/tar.ts Outdated Show resolved Hide resolved

ethanis added 6 commits March 17, 2020 16:42

Clean import orders

3e55aff

Merge master into ethanis/cache-multiple-paths

f68f5d0

Use fs promises in actionUtils tests

6563669

Update import order for to fix linter errors

af3e500

Fix remaining linter error

bc1250c

Remove platform-specific test code

b862af8

dhadka approved these changes Mar 19, 2020

View reviewed changes

boredland approved these changes Mar 20, 2020

View reviewed changes

Add lerna example for caching multiple dirs

482e3f4

joshmgross approved these changes Mar 20, 2020

View reviewed changes

examples.md Outdated Show resolved Hide resolved

Lerna example updated to v2

8c1c771

Co-Authored-By: Josh Gross <joshmgross@github.com>

joshmgross merged commit eb78578 into actions:master Mar 20, 2020

This was referenced Mar 20, 2020

Use zstd instead of gzip #202

Closed

multiple jobs: Cache not found for input keys #225

Closed

marcinczenko mentioned this pull request Mar 24, 2020

ci: enables caching dependencies identity-box/identity-box#33

Merged

ethanis mentioned this pull request Mar 25, 2020

Multiple directories and path glob pattern support for pipeline caching microsoft/azure-pipelines-agent#2834

Merged

joshmgross mentioned this pull request Apr 2, 2020

Fix Lerna Example #242

Merged

dhadka mentioned this pull request Apr 26, 2020

Any way to force overwrite cache? #274

Closed

dhadka mentioned this pull request May 12, 2020

Documentation on multiple caches in a single script #288

Closed

This was referenced May 22, 2020

Cache single file #33

Closed

Cache multiple paths #16

Closed

coderabbitai bot mentioned this pull request Mar 18, 2026

Add dependency review workflow to scan for license violations and CVEs AOSSIE-Org/Template-Repo#105

Merged

4 tasks

Conversation

ethanis commented Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhadka commented Mar 12, 2020

Uh oh!

ethanis commented Mar 12, 2020

Uh oh!

ethanis Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

dhadka Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

ethanis Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

joshmgross Mar 17, 2020

Choose a reason for hiding this comment

Uh oh!

smorimoto commented Mar 17, 2020

Uh oh!

smorimoto commented Mar 17, 2020

Uh oh!

dhadka commented Mar 17, 2020

Uh oh!

smorimoto commented Mar 17, 2020

Uh oh!

joshmgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ethanis commented Mar 19, 2020

Uh oh!

boredland commented Mar 19, 2020

Uh oh!

dhadka left a comment

Choose a reason for hiding this comment

Uh oh!

joshmgross commented Mar 20, 2020

Uh oh!

chrispat commented Mar 20, 2020

Uh oh!

boredland commented Mar 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethanis commented Mar 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smorimoto commented Mar 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethanis commented Mar 20, 2020

Uh oh!

joshmgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gcourtemanche commented Apr 2, 2020

Uh oh!

joshmgross commented Apr 2, 2020

Uh oh!

Bnaya commented Apr 25, 2020

Uh oh!

joshmgross commented Apr 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ethanis commented Mar 12, 2020 •

edited

Loading

boredland commented Mar 20, 2020 •

edited

Loading

ethanis commented Mar 20, 2020 •

edited

Loading

smorimoto commented Mar 20, 2020 •

edited

Loading