Revamp GitHub Actions workflow for SQL dump and GeoPackages upload#41
Merged
anthonyfok merged 8 commits intomasterfrom Mar 29, 2022
Merged
Revamp GitHub Actions workflow for SQL dump and GeoPackages upload#41anthonyfok merged 8 commits intomasterfrom
anthonyfok merged 8 commits intomasterfrom
Conversation
To be replace with more lightweight workflows, see #36
This replaces gpkg-to-pgdump.yml with the following changes: - opendrr-boundaries.sql in the git repo is used directly - opendrr-boundaries.dump is no longer built in the workflow - opendrr-boundaries.sql is split into 2GB-chunks before uploading as release assets (#34) - No longer uploads to Amazon S3 and Backblaze B2 buckets - GeoPackages (*.gpkg) are no longer uploaded by this workflow; they will be handled by a new upload-geopackages.yml instead Fixes #34 See also #36
Downloading a large GitHub artifact can sometimes be very slow, with a 3GB artifact taking over 45 minutes, so keeping a copy on a Backblaze B2 bucket allows faster download if necessary.
This uploads *.gpkg files as release assets as the old gpkg-to-pgdump.yml workflow used to do. As softprops/action-gh-release@v1 currently fails to upload all 6.0 GiB of GeoPackages at once, xresloader/upload-to-github-release@v1 is now used for the task. See #36
This workflow prevents the checked-out Git LFS files in the workflow cache from getting stale and removed by accessing the cache every Sunday at 00:38 or 01:38 Pacific Time.
Upstream has updated from Node 12 to Node 16 in these GitHub Actions.
Note: actions/cache v3.0.0 has trouble restoring cache over
2 GiB in size:
Warning: The value of "length" is out of range.
It must be >= 0 && <= 2147483647. Received 2419791709
so the upgrade to actions/cache@v3 will have to wait until the
resolution of Issue actions/cache#773 in PR actions/cache#775.
into new file upload-sqldump-and-geopackages.yml.
The 'prevent-race-condition' concurrency is moved from job level
to workflow level in order to prevent workflow cancellation:
Canceling since a higher priority waiting request
for 'prevent-race-condition' exists
Proper workflow_dispatch is also implemented in case there is a need to
manually (re)upload the release assets.
Together with the previous commits, the refactoring of the previous
gpkg-to-pgdump.yml GitHub Actions workflow is now complete.
Fixes #36
wkhchow
approved these changes
Mar 28, 2022
Collaborator
wkhchow
left a comment
There was a problem hiding this comment.
Looks good to me! Will update asset release after PR merged. Will need to rebase changes to current working branch (test_hexbin_unclipped) and PR/merge for another release.
Member
Author
|
Thank you @wkhchow for your review! I will merge this into the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces gpkg-to-pgdump.yml (Convert GeoPackage files to PostGIS archive) GitHub Actions workflow with upload-sqldump-and-geopackages.yml. Changes include:
workflow_dispatchmanual trigger for re-upload to specified released tag.Also, a new refresh-cache.yml workflow is added:
concurrency(which I namedprevent-race-condition) is added to prevent upload-sqldump-and-geopackages.yml and refresh-cache.yml workflow from trying to write to the cache at the same time.Tested at https://github.com/anthonyfok/boundaries/actions
Fixes #34, fixes #36