Skip to content

Use zstd instead of gzip#202

Closed
smorimoto wants to merge 8 commits intoactions:masterfrom
smorimoto:zstd
Closed

Use zstd instead of gzip#202
smorimoto wants to merge 8 commits intoactions:masterfrom
smorimoto:zstd

Conversation

@smorimoto
Copy link
Copy Markdown
Contributor

@smorimoto smorimoto commented Feb 28, 2020

Closes: #184 #139
Useful benchmark and explanation: #6 (comment)

I would like to thank the following people for their help in making this change (in no particular order): @joshmgross @kcgen @hugovk @miketimofeev

@smorimoto
Copy link
Copy Markdown
Contributor Author

I use my minimal zstd docker image for testing, but we can copy and manage it under the Actions organization, or I can invite you as collaborators. (If you want.)

@smorimoto
Copy link
Copy Markdown
Contributor Author

For now, this seems to be working for us. Thank you so much to the many people who worked so hard for this change.

@smorimoto
Copy link
Copy Markdown
Contributor Author

@joshmgross @dhadka This is ready for review. Please review this when you have time. and I will leave it to you when to merge this.

@smorimoto
Copy link
Copy Markdown
Contributor Author

At first, I thought that there might be some problems for the users using the master branch, but that seems to be no problem.

@boredland
Copy link
Copy Markdown

this cuts our caching speed in half, even though its still enormous (400mb node_modules from a yarn mongorepo. 33-50s to download/extract, 40s to copy the node_modules dirs into the right places)

@smorimoto
Copy link
Copy Markdown
Contributor Author

@boredland By the way, this is a bit off-topic, but caching node_modules is a bad idea.

@smorimoto
Copy link
Copy Markdown
Contributor Author

and this is a change for the purpose of speed up cache action using zstd, so it's not a topic to discuss here even if the result is still slow.

@joshmgross
Copy link
Copy Markdown
Contributor

Just merged #212, so there are a bunch of merge conflicts.

Additionally, now that we're including version in the cache that we create, I think we should support both tar and zstd and include that information in the version. Self-hosted runners might not have zstd installed.

Let me know if you need any help or want me to take over this PR

@smorimoto
Copy link
Copy Markdown
Contributor Author

@joshmgross Well, I wasn't used to the test with Jest and I couldn't write very clean. I don't think there will be much change, so can you push some commits to this branch?

@boredland
Copy link
Copy Markdown

Hi, are there any updates? I really like the speed with this.

@joshmgross
Copy link
Copy Markdown
Contributor

@boredland We're still planning on making this change, just need to ensure that we have a good scenario for self-hosted runners that might not have zstd installed

@smorimoto
Copy link
Copy Markdown
Contributor Author

This PR is probably never merged. Because @aiqiaoy took over the rest of this PR work from me. Please check the new PR for additional information.

@smorimoto
Copy link
Copy Markdown
Contributor Author

I will keep this PR as a draft until the new PR is merged, but I will close it when it's merged.

@smorimoto smorimoto closed this May 4, 2020
@smorimoto smorimoto deleted the zstd branch May 4, 2020 14:51
@smorimoto
Copy link
Copy Markdown
Contributor Author

This was merged into the master in another PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use zstd instead of gzip

3 participants