Skip to content

Manage package info as JSON#27

Merged
taiki-e merged 2 commits intomainfrom
json
Dec 24, 2022
Merged

Manage package info as JSON#27
taiki-e merged 2 commits intomainfrom
json

Conversation

@taiki-e
Copy link
Owner

@taiki-e taiki-e commented Oct 4, 2022 •
edited
Loading

Summary

  • Verify SHA256 checksums for downloaded files in all tools installed from GH releases to provide a higher level of security
  • Automate update of the @latest version in all tools installed from GH releases
  • Support omitting the patch/minor version in all tools installed from GH releases
  • Add guide about adding new tools.

Closes #1
Closes #5
Closes #25

TODO:

  • should we use sha256 or sha512/256 instead of sha512 to reduce the size of the manifest files? UPD: Changed to use sha256 as it seems to reduce the overall size of the manifest by almost half.
  • To reduce the size of the manifest, we may want to avoid generating manifests for very old versions or test releases.
    • We have added the ability to do this, but have not applied it to the currently supported tools.

UPDATE: manifest size issue is currently tracked by #36.

Details

This switches package info management from code in shellscript to JSON manifest. The following is an example of the manifest:

install-action/manifests/mdbook.json

Lines 2 to 20 in f30d182

"latest": {
"version": "0.4.25",
"x86_64_linux_musl": {
"url": "https://github.com/rust-lang/mdBook/releases/download/v0.4.25/mdbook-v0.4.25-x86_64-unknown-linux-musl.tar.gz",
"checksum": "f8481706ceda01d170aa02e1d87cf61abb24e5b67f9081f3d306f2861f1af5aa"
},
"x86_64_macos": {
"url": "https://github.com/rust-lang/mdBook/releases/download/v0.4.25/mdbook-v0.4.25-x86_64-apple-darwin.tar.gz",
"checksum": "5f11014060f557919c5abc44f051d08f08f4ed036d1086b06b5dcdcea1ced64f"
},
"x86_64_windows": {
"url": "https://github.com/rust-lang/mdBook/releases/download/v0.4.25/mdbook-v0.4.25-x86_64-pc-windows-msvc.zip",
"checksum": "771e5aa0bffbb475a05f8fa1889550475672fbbec229bc1cfa85e16c3c8852c8"
},
"aarch64_linux_musl": {
"url": "https://github.com/rust-lang/mdBook/releases/download/v0.4.25/mdbook-v0.4.25-aarch64-unknown-linux-musl.tar.gz",
"checksum": "6a49db5a2681c485b59d870d309364537a8d9f646b6f22cce684794f4ea05c0d"
}
},

These manifests are generated by a manifest generator written in Rust. The generator uses base package info like the following, fetches Release info from GitHub API, finds releases that include specified assets, downloads assets, gets SHA256 checksums, and dumps these info as manifests.

install-action/tools/codegen/base/mdbook.json

Lines 1 to 14 in f30d182

{
"repository": "https://github.com/rust-lang/mdBook",
"tag_prefix": "v",
"asset_name": "${package}-v${version}-${rust_target}.tar.gz",
"platform": {
"x86_64_linux_musl": {},
"x86_64_linux_gnu": {},
"x86_64_macos": {},
"x86_64_windows": {
"asset_name": "${package}-v${version}-${rust_target}.zip"
},
"aarch64_linux_musl": {}
}
}

Multiple asset name patterns can be specified, and if no assets match the first pattern, check the second pattern. So it works well even if the asset name changes in a new release (just insert the new asset name to the front of the asset name list). The following is the case of cross, which changed asset names in 0.2.2.

install-action/tools/codegen/base/cross.json

Lines 4 to 7 in f30d182

"asset_name": [
"${package}-${rust_target}.tar.gz",
"${package}-v${version}-${rust_target}.tar.gz"
],

install-action/manifests/cross.json

Lines 62 to 91 in f30d182

"0.2.2": {
"version": "0.2.2",
"x86_64_linux_musl": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.2/cross-x86_64-unknown-linux-musl.tar.gz",
"checksum": "e31df42dc18659ef3caf0f6b41a8fabb0c7356ba95c87516625271791a113439"
},
"x86_64_macos": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.2/cross-x86_64-apple-darwin.tar.gz",
"checksum": "c0d5b4269a5954211e0893c9c519917b0b6cd1c7c8ac9c1881341d3e10d41e1e"
},
"x86_64_windows": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.2/cross-x86_64-pc-windows-msvc.tar.gz",
"checksum": "216c5f19cb8ceff571fa29c6876c8d39b672739f45edc1046289e31049a58c4f"
}
},
"0.2.1": {
"version": "0.2.1",
"x86_64_linux_musl": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.1/cross-v0.2.1-x86_64-unknown-linux-musl.tar.gz",
"checksum": "b2e2ff0c25cb1787c0e44984136c2243b14df20023adcc7cfb9170ecfde7ad68"
},
"x86_64_macos": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.1/cross-v0.2.1-x86_64-apple-darwin.tar.gz",
"checksum": "589da89453291dc26f0b10b521cdadb98376d495645b210574bd9ca4ec8cfa2c"
},
"x86_64_windows": {
"url": "https://github.com/cross-rs/cross/releases/download/v0.2.1/cross-v0.2.1-x86_64-pc-windows-msvc.tar.gz",
"checksum": "3af59ff5a2229f92b54df937c50a9a88c96dffc8ac3dde520a38fdf046d656c4"
}
},

The generator is run regularly in CI, and a PR is created if there is a release that does not yet exist in the manifest.

@taiki-e taiki-e force-pushed the json branch 2 times, most recently from 39e4fce to 32ee3ca Compare October 4, 2022 01:56
@taiki-e taiki-e force-pushed the json branch 3 times, most recently from bbc09b4 to d963946 Compare October 18, 2022 12:45
@taiki-e taiki-e marked this pull request as ready for review October 18, 2022 12:46
@taiki-e taiki-e force-pushed the json branch 10 times, most recently from 7dfd3b7 to 46545b6 Compare October 24, 2022 15:42
@taiki-e taiki-e force-pushed the json branch 2 times, most recently from 43d6048 to 1e961e0 Compare October 28, 2022 13:06
@taiki-e taiki-e force-pushed the json branch 4 times, most recently from dad4550 to f653545 Compare December 3, 2022 22:11
@taiki-e taiki-e force-pushed the json branch 8 times, most recently from 094a030 to a66f899 Compare December 11, 2022 07:16
@taiki-e taiki-e force-pushed the json branch 15 times, most recently from 2a01847 to 5be31fd Compare December 17, 2022 16:39
@taiki-e taiki-e force-pushed the json branch 3 times, most recently from 277318e to b8a682b Compare December 22, 2022 02:36
@taiki-e taiki-e mentioned this pull request Dec 22, 2022
Closed
@taiki-e taiki-e force-pushed the json branch 3 times, most recently from 4311a19 to ace4f84 Compare December 24, 2022 05:04
@taiki-e taiki-e merged commit f30d182 into main Dec 24, 2022
@taiki-e taiki-e deleted the json branch December 24, 2022 12:53
This was referenced Dec 24, 2022
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support omitting patch/minor version in @version syntax Auto-detecting latest tool versions Checksum

1 participant

@taiki-e