Introduce the dependency caching for Maven and Gradle #193

KengoTODA · 2021-07-13T01:37:35Z

This PR suggests the dependency caching feature just like the dependency caching feature of actions/setup-node.

Benefits:

It makes the workflow YAML file shorter, no need to run actions/cache explicitly any more.
It can run Java installation and cache restoring process in parallel, so elapsed time for initialization will be shorter.

About the performance, we can expect that it'll shorten about 28% of the time to build, even when we have no dependency. You can find the code to run the test in my test project, or see setup-java-cache.csv or Google Docs to grab detailed data.

Note that in this graph I run 10 tests and use 9 of them (treated 1 of them as error) for each dependency size.

Considerations:

This feature does not provide the upload-chunk-size configuration. I think it won't be problem for most users, and actions/setup-node also provides no config like this.
We may support other package managers like ivy and grape, but first let's focus on package managers described in the doc of actions/cache.

Check list:

Mark if documentation changes are required.
Mark if tests were added or updated to cover the changes.

Thanks in advance for your review! 😃

maxim-lobanov · 2021-07-13T10:58:28Z

Hello @KengoTODA , thank you for contribution!
Your changes look really awesome. Please give us some time to review it properly and make sure that everything is okay and we will be happy to accept it.

src/setup-java.ts

src/cache.ts

.github/workflows/e2e-cache.yml

https://github.com/actions/setup-java/pull/193/files#r669521434

KengoTODA · 2021-07-14T12:38:21Z

@dmitry-shibanov I've updated the code, please check.

I also found that the Gradle cache on Windows isn't be saved as expected, see this build result for detail. I will investigate.

dmitry-shibanov · 2021-07-14T13:26:13Z

@dmitry-shibanov I've updated the code, please check.

I also found that the Gradle cache on Windows isn't be saved as expected, see this build result for detail. I will investigate.

Hello @KengoTODA. Thank you for your help. It looks like a known issue with gradle on windows-latest. Possibly it will help: actions/cache#454

morki · 2021-07-14T15:56:53Z

Hi, I see the usage in some test like this:

- uses: actions/setup-java@v2
  with:
    distribution: 'adopt'
    java-version: '11'
    cache: 'gradle' # will restore cache of dependencies and wrappers

Please will it be possible to use working-directory to make it use other Gradle directory than root like this?

- uses: actions/setup-java@v2
  working-directory: server
  with:
    distribution: 'adopt'
    java-version: '11'
    cache: 'gradle' # will restore cache of dependencies and wrappers

We use monorepo structure, where server directory is for Java application and client for Node one.
Maybe some other solution can be used to accomplish this.

Thank you very much for such a great feature!

KengoTODA · 2021-07-15T01:19:57Z

Thanks all,

I've finished the necessary improvements, please check updates here.
Now e2e tests on Windows are stable, we also have tests for Maven, and the restore process runs after the auth part.

src/cache.ts

…indows

KengoTODA · 2021-07-20T04:00:47Z

To make the permission trouble on Windows more intuitive, added a warning like below:

Refer to the demo in my repo if necessary.

maxim-lobanov · 2021-08-02T08:15:27Z

src/cache.ts

+  core.debug(`primary key is ${primaryKey}`);
+  core.saveState(STATE_CACHE_PRIMARY_KEY, primaryKey);
+  if (primaryKey.endsWith('-')) {
+    core.warning(


I wonder if we would like to throw new Error() if no files are found matched patterns like we have done in setup-node.

@AlenaSviridenko @MaksimZhukov what do you think about it?

If we would like to keep it as a warning, I think we will need one more condition in save function because current implementation will try to save cache even in case when no files are matched patterns. I guess we don't want it.

Thanks for your review! I just kept the existing behaviour (@actions/cache with hashFiles(...)), but it would be a great improvement making cache more intuitive. 👍

I'll wait for the reply from other experts, and change the code once we made a consensus.

@konradpabjan for any thoughts

I'm leaning towards this throwing an error if no files are found.

Given that the new cache input is optional, it's reasonable to expect that someone who specifies gradle or maven has the appropriate files in their repository.

A lot of users today also incorrectly use actions/cache for a variety of reasons an an error would be pretty explicit in telling users to fix their code or not to use the cache feature.

I made a proposal in my repo: https://github.com/KengoTODA/setup-java/pull/4/files

If it looks good to you, I'll merge this commit to this PR. :)

@KengoTODA , Proposal looks good to me. Please merge PR.

Also, I see that two checks fail right now:

license - please ignore this check for now. I will fix it in the separate PR.

build fails on verify_no_unstaged_changes - may be you haven't rebuild it last time? Let's try to rebuild one more time after merging your proposal via npm run build && npm run release

src/cleanup-java.ts

maxim-lobanov · 2021-08-02T08:24:29Z

@KengoTODA, thank you for your contribution. I have left a few minor comments.
There is one more issue that we need to resolve before this pull-request can be merged:
Post-job step does two action: clean up auth and save cache.
Post-job condition is always() currently. It means that post-job step will be run even job is failed. It is necessary because auth data should be clean up always.
The problem is that we don't want to save cache if job fails because obviously, it will cause invalid caching and save and etc (for example, when gradle install fails).
I didn't find the way to determine if the whole job is successful or not in post job step. Will try to ask our engineers if there is a way to overcome it.

Please let me know if you have any suggestions / concerns here.

dmitry-shibanov · 2021-08-17T15:30:21Z

Hello @KengoTODA. Sorry for the late reply. Could you please take a look on pull request we've created in your repository ? We've added logic to save cache only for successful builds.
Besides, could you please pull latest main to resolve the conflict ?

maxim-lobanov · 2021-08-18T11:22:39Z

@KengoTODA, sorry for delay and thank you for your patience. Issue with post-job required to apply "hacks" :)
I think we are on the final line with this PR and it will be merged soon. We will double validate from our side and engage a few more experts to review.

konradpabjan

Awesome PR and thank you for the contribution! 🎉

Left a few minor comments.

Edit:

The changes to the index.js files under /dist are also huge (pretty much impossible to manually verify). I've created a separate PR to add a automated check to make it easier to verify the files are correct. Want to do a bit more verification once #206 goes in.

konradpabjan · 2021-08-18T23:55:04Z

src/cache.ts

+  core.debug(`primary key is ${primaryKey}`);
+  core.saveState(STATE_CACHE_PRIMARY_KEY, primaryKey);
+  if (primaryKey.endsWith('-')) {
+    core.warning(


I'm leaning towards this throwing an error if no files are found.

Given that the new cache input is optional, it's reasonable to expect that someone who specifies gradle or maven has the appropriate files in their repository.

A lot of users today also incorrectly use actions/cache for a variety of reasons an an error would be pretty explicit in telling users to fix their code or not to use the cache feature.

src/cleanup-java.ts

src/cache.ts

src/util.ts

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

it still has three errors as follows: >* setup-java.npm.sax > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/sax.dep.yml > - license needs review: other > >* setup-java.npm.tslib-1.14.1 > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-1.14.1.dep.yml > - license needs review: 0bsd > >* setup-java.npm.tslib-2.3.0 > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-2.3.0.dep.yml > - license needs review: 0bsd

KengoTODA · 2021-08-19T11:30:04Z

I've updated licensed files with licensed v3.1.0. It still has three status check failures:

setup-java.npm.sax
filename: /Users/kengo/GitHub/setup-java/.licenses/npm/sax.dep.yml

license needs review: other

setup-java.npm.tslib-1.14.1
filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-1.14.1.dep.yml

license needs review: 0bsd

setup-java.npm.tslib-2.3.0
filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-2.3.0.dep.yml

license needs review: 0bsd

The license of sax v1.2.4 is ISC. And all of them are already used in the setup-node action, so I hope there is no problem using them in this project:

maxim-lobanov · 2021-08-19T11:33:23Z

@KengoTODA ,
As for the licenses:

Change sax.dep.yml to set license: isc manually.
Add 0bsd to https://github.com/actions/setup-java/blob/main/.licensed.yml#L4 as it is done for setup-node: https://github.com/actions/setup-node/blob/v2.4.0/.licensed.yml

Could you please also merge your proposal KengoTODA#4 and npm run build && npm run release after that?

actions#193 (comment)

KengoTODA · 2021-08-19T12:15:48Z

Here is (probably) the final performance comparison chart. As expected, the performance is better than existing way:

Source (workflow runs): large, medium, and small

konradpabjan

🚀 🥳 LGTM!

maxim-lobanov · 2021-08-19T17:20:47Z

@KengoTODA thank you one more time for great contribution, very appreciate!
I have merged PR and changes are already in main branch.
I will cut the new action release on Monday (we are trying to avoid releases on Friday)

KengoTODA · 2021-08-31T01:21:56Z

I'm very glad to see a changelog entry announcing this change. Thanks again everyone! 🚀

jackromo888

https://github.com/oracle-actions/setup-java.wiki.git

* implement a core logic to cache dependnecies * integrate the cache logic to entry points * add a user doc about the dependency cache feature * reflect changes to the dist dir * add a prefix to the cache key https://github.com/actions/setup-java/pull/193/files#r669521434 * test: extract build.gradle to a file in __tests__ dir * run the restore e2e test on the specified OS * add an e2e test for maven * fix the dependency among workflows * stabilize the cache on the Windows in e2e test * add .gitignore files to __tests__/cache directories * try to run restore after the authentication * use the key in state to save caches in the post process * suggest users to run without daemon if fail to save Gradle cache on Windows * add missing description in the README.md * run clean-up tasks in serial * Add validation for post step (actions#3) * work on fixing cache post step * fix tests * Update src/cleanup-java.ts Co-authored-by: Konrad Pabjan <konradpabjan@github.com> * Update src/cache.ts Co-authored-by: Konrad Pabjan <konradpabjan@github.com> * style: put the name of input to the constants.ts * format: run `npm run build` to reflect changes to the dist dir * chore: update licensed files by `licensed cache` it still has three errors as follows: >* setup-java.npm.sax > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/sax.dep.yml > - license needs review: other > >* setup-java.npm.tslib-1.14.1 > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-1.14.1.dep.yml > - license needs review: 0bsd > >* setup-java.npm.tslib-2.3.0 > filename: /Users/kengo/GitHub/setup-java/.licenses/npm/tslib-2.3.0.dep.yml > - license needs review: 0bsd * fix: rerun ncc on macOS with node v12 * build: follow the suggestion at PR page actions#193 (comment) * fix: throw error in case of no package manager file found Co-authored-by: Dmitry Shibanov <dmitry-shibanov@github.com> Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

KengoTODA added 4 commits July 13, 2021 09:04

implement a core logic to cache dependnecies

fef7d58

integrate the cache logic to entry points

4a7bf99

add a user doc about the dependency cache feature

945940e

reflect changes to the dist dir

0591f86

dmitry-shibanov reviewed Jul 14, 2021

View reviewed changes

KengoTODA added 5 commits July 14, 2021 19:50

add a prefix to the cache key

f537562

https://github.com/actions/setup-java/pull/193/files#r669521434

test: extract build.gradle to a file in __tests__ dir

b047f97

run the restore e2e test on the specified OS

3e2fde2

add an e2e test for maven

a5a0c52

fix the dependency among workflows

1872d8e

KengoTODA added 3 commits July 15, 2021 08:29

stabilize the cache on the Windows in e2e test

313e1ad

add .gitignore files to __tests__/cache directories

8687e45

try to run restore after the authentication

6977c03

dmitry-shibanov reviewed Jul 15, 2021

View reviewed changes

src/cache.ts Outdated Show resolved Hide resolved

KengoTODA added 3 commits July 16, 2021 09:22

use the key in state to save caches in the post process

7fe6c4d

suggest users to run without daemon if fail to save Gradle cache on W…

fae2927

…indows

add missing description in the README.md

f6a3b97

dmitry-shibanov approved these changes Jul 29, 2021

View reviewed changes

maxim-lobanov reviewed Aug 2, 2021

View reviewed changes

src/cleanup-java.ts Outdated Show resolved Hide resolved

run clean-up tasks in serial

5f3f74c

KengoTODA mentioned this pull request Aug 12, 2021

Housekeeping: Update zeit/ncc to vercel/ncc #190

Closed

2 tasks

maxim-lobanov approved these changes Aug 18, 2021

View reviewed changes

vsafonkin approved these changes Aug 18, 2021

View reviewed changes

konradpabjan reviewed Aug 19, 2021

View reviewed changes

KengoTODA and others added 4 commits August 19, 2021 10:28

Update src/cleanup-java.ts

098a656

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

Update src/cache.ts

cb966d0

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

style: put the name of input to the constants.ts

6edf849

format: run npm run build to reflect changes to the dist dir

0082baa

KengoTODA mentioned this pull request Aug 19, 2021

fix: throw error in case of no package manager file found KengoTODA/setup-java#4

Closed

KengoTODA added 2 commits August 19, 2021 19:19

fix: rerun ncc on macOS with node v12

807e574

KengoTODA added 2 commits August 19, 2021 19:51

build: follow the suggestion at PR page

33dfe17

actions#193 (comment)

fix: throw error in case of no package manager file found

67b74a1

konradpabjan approved these changes Aug 19, 2021

View reviewed changes

maxim-lobanov merged commit 08e4e81 into actions:main Aug 19, 2021

KengoTODA deleted the introduce-cache branch August 19, 2021 23:34

KengoTODA mentioned this pull request Aug 26, 2021

Bump up actions/setup-java to speed up the caching spotbugs/spotbugs#1668

Merged

steven-sheehy mentioned this pull request Aug 31, 2021

Switch to actions/setup-java's built-in cache hiero-ledger/hiero-mirror-node#2481

Merged

2 tasks

This was referenced Sep 3, 2021

Update actions in workflows jspecify/jspecify#212

Merged

feat(cache): change key to match actions/cache documentation #213

Closed

docs: Append --no-daemon option to dependencies cache with Gradle #218

Merged

KengoTODA mentioned this pull request Dec 5, 2021

Use Gradle 7 to build the project WorksApplications/elasticsearch-sudachi#79

Merged

jackromo888 reviewed Aug 9, 2022

View reviewed changes

hannesj mentioned this pull request Oct 24, 2022

Update GH Actions versions opentripplanner/OpenTripPlanner#4542

Merged

Introduce the dependency caching for Maven and Gradle #193

Introduce the dependency caching for Maven and Gradle #193

Uh oh!

Conversation

KengoTODA commented Jul 13, 2021

Uh oh!

maxim-lobanov commented Jul 13, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KengoTODA commented Jul 14, 2021

Uh oh!

dmitry-shibanov commented Jul 14, 2021

Uh oh!

morki commented Jul 14, 2021

Uh oh!

KengoTODA commented Jul 15, 2021

Uh oh!

Uh oh!

KengoTODA commented Jul 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxim-lobanov Aug 2, 2021

Choose a reason for hiding this comment

Uh oh!

KengoTODA Aug 2, 2021

Choose a reason for hiding this comment

Uh oh!

maxim-lobanov Aug 18, 2021

Choose a reason for hiding this comment

Uh oh!

konradpabjan Aug 18, 2021

Choose a reason for hiding this comment

Uh oh!

KengoTODA Aug 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxim-lobanov Aug 19, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maxim-lobanov commented Aug 2, 2021

Uh oh!

dmitry-shibanov commented Aug 17, 2021

Uh oh!

maxim-lobanov commented Aug 18, 2021

Uh oh!

konradpabjan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

konradpabjan Aug 18, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KengoTODA commented Aug 19, 2021

Uh oh!

maxim-lobanov commented Aug 19, 2021

Uh oh!

KengoTODA commented Aug 19, 2021

Uh oh!

konradpabjan left a comment

Choose a reason for hiding this comment

Uh oh!

maxim-lobanov commented Aug 19, 2021

Uh oh!

KengoTODA commented Aug 31, 2021

Uh oh!

jackromo888 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

KengoTODA commented Jul 20, 2021 •

edited

Loading

KengoTODA Aug 19, 2021 •

edited

Loading

konradpabjan left a comment •

edited

Loading