What are the actual problems of GH Actions?

Well, here's a short list which affect umoci:

The documentation is lacking. It took me a fair while to configure the CI, mostly because the documentation (in true GitHub fashion) doesn't provide a proper reference text for the features and semantics. You have to carefully read the getting started sections as well as the "reference" text to figure out what is going on -- and many parts of the reference text completely omit information that is pretty important (for instance, the documentation on ${{ }} doesn't describe what many of the built-in contexts mean).
- This is compounded by the fact that many incredibly basic features like "check out the code of this repository" are not extensively documented by the Workflows documentation, so you have to look through the https://github.com/actions/checkout README in order to understand what all the parameters do (and even then the README is missing information as well).
- Lots of the documentation links take you to the GitHub API documentation which doesn't actually answer what the link says (for instance, try to figure out what each of the possible event types mean in the on block -- all of the links take you to the GitHub API which is written from the perspective of a GitHub App and the set of events in the API are different to the ones available in Workflows).
Performance (both start-up time as well as general performance) is worse than Travis in my experience. Right now the test umoci CI I have runs far longer than Travis (28-34 minutes versus >40 minutes) and the GitHub Actions configuration I have now uses cached build images -- when I disabled caching it took significantly more than an hour.
The caching feature is of questionable utility since it seems like it cannot use caches uploaded during the current workflow run -- the workflow has to be finished before future jobs can make use of the cache. This means that even if a job hypothetically manages to build the image and upload it to the cache (which you cannot do manually because the cache action does uploads at the end of a job) all other jobs will repeat the same work.
- I use the word "seems" because whether this is true is both not documented and the documentation lives in the README of the Node.js library which the Actions is implemented using. It's possible that I'm wrong, but some test runs seem to show that this is the case.
The only way of transferring information between jobs is using artefacts (for umoci this is used for the coverage profile collation -- where all I want is to have a final fan-in job which runs an awk and shell script on some text files generated by each job), and this feature has an abundance of issues:
- As before, the documentation is in a GitHub README.
- You can upload to the same artefact name more than once, but doing so results in you modifying the artefact (not replacing its contents) and this feature is explicitly warned against because it can cause "corruption" (though I assume they mean there's a race condition where you can lose files added by another job). I do not understand why they added this feature if it doesn't work by design. This means you have to give a unique name to each artefact.
- Artefacts generated during a workflow cannot be deleted during that workflow's execution (funnily enough you couldn't delete them at all until a few months ago), meaning that all of these artefacts will be displayed for download at the end of the workflow run. There are workarounds (where you have an action that creates a new workflow and triggers it after the current workflow is finished) but they would certainly cause more issues. This isn't too bad until you consider that:
  - Artefacts do not expire (or at least, they didn't according to the GitHub issues -- this is not documented anywhere -- but the GitHub API appears to have some expiry fields which I don't understand).
  - There is a per-repository limit of how much total storage can be used by artefacts, at which point uploads will fail or you will start getting charged money.

Don't get me wrong, we could just use it to run make ci. It's just that I think it's not suited to anything more complicated than that, given that all I wanted to do was to be able to stop duplicating the make unit and make ci-image steps as well as collate the code coverage over all test runs across operating systems and I ended up with a setup which seems to be slower despite doing less work. My goal was to eventually have fuzzing jobs for umoci which would dynamically update the fuzzing corpus as part of a scheduled GitHub Action -- but it seems that this would just be even more painful.

It doesn't support nested VM AFAIK.

As in vagrant doesn't work, or it's slow? (Though I guess either way that is a problem.) Does this apply to the "machine" executor?

Also, its free(-as-in-beer) edition has tight limitation: https://circleci.com/pricing/#comparison-table

Yeah I found this out as well, however since we are an open source project we qualify for more features. For umoci, the need to email them to get MacOS support is a bit odd but that doesn't affect runc.

CI: move to GitHub Actions? #2512

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions