Skip to content

Github Actions workflow for building the toolchain and the SDK#227

Merged
Vogtinator merged 23 commits into
ndless-nspire:masterfrom
shinyblink:ndless-ci
Oct 17, 2020
Merged

Github Actions workflow for building the toolchain and the SDK#227
Vogtinator merged 23 commits into
ndless-nspire:masterfrom
shinyblink:ndless-ci

Conversation

@fridtjof

@fridtjof fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor

My original motivation for this was to let you provide "nightly" builds of the SDK and toolchain, so our CI would not have to duplicate effort, but it seems that GitHub Actions does not let you easily download artifacts from other repositories (yet?).

Still, the effort was made, so here's a free CI workflow for your project! :)

On runtimes:

  • The toolchain takes roughly 40 minutes to build. Caching is implemented, and a rebuild is triggered whenever the build-toolchain.sh script changes (because it contains the gcc, binutils, etc version numbers used to build the toolchain). Unfortunately, jobs can't access past run's artifacts, so that has to uploaded on every run, adding ~2 min of runtime :/ A possible (but less clean) solution would be to merge the two jobs into one.
  • Caching does not make sense for the SDK and other builds in this repo, because they obviously change all the time.

(note that this PR does not activate CI yet, for security reasons)

@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

Actually, there seems to be a problem with file permissions. I will fix this later today.

@fridtjof fridtjof marked this pull request as draft August 4, 2020 13:28
@fridtjof fridtjof marked this pull request as ready for review August 4, 2020 14:14
@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

Testing on the fork has concluded, this workflow is now green and ready for review

@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

I think I'll add macOS CI in another PR after this one.

Definitely feel free to squash all these commits, they turned out to be more than I anticipated.

@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

Might as well add ubuntu-20.04 too.

Good point, will do!

@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

It's included now, CI is testing here: https://github.com/shinyblink/Ndless/runs/945763481?check_suite_focus=true
Currently going through a full toolchain build for all three platforms (in parallel), should be done in ~45min

@fridtjof

fridtjof commented Aug 4, 2020

Copy link
Copy Markdown
Contributor Author

Should be ready for review now. Really this time.

@Vogtinator Vogtinator left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I don't really like this kind of CI as it's entirely proprietary and mostly just vendor lock-in, so I never looked into it myself. This looks simple and maintainable enough though, that it' probably worth it.

Except for make -j4 no huge blocking issues.

(note that this PR does not activate CI yet, for security reasons)

I'm curious, what are those security reasons?

Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
@fridtjof

fridtjof commented Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

I'm curious, what are those security reasons?

Runners have access to tokens that allow you to do various privileged things to the repository.
To avoid me (or anyone else) from just extracting them with a malicious workflow submitted through a PR, CI does not run on PRs that originate from forks.

@fridtjof

fridtjof commented Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

Now building: https://github.com/shinyblink/Ndless/actions/runs/196238815

This includes another full toolchain rebuild, because the cache paths have changed (now also includes the toolchain install directory so we can use it directly).

@adriweb

adriweb commented Aug 5, 2020

Copy link
Copy Markdown
Contributor

I'm curious, what are those security reasons?

Runners have access to tokens that allow you to do various privileged things to the repository.
To avoid me (or anyone else) from just extracting them with a malicious workflow submitted through a PR, CI does not run on PRs that originate from forks.

Note that this got improved 2 days ago :) https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/

@Vogtinator

Copy link
Copy Markdown
Contributor

I'm curious, what are those security reasons?

Runners have access to tokens that allow you to do various privileged things to the repository.
To avoid me (or anyone else) from just extracting them with a malicious workflow submitted through a PR, CI does not run on PRs that originate from forks.

Note that this got improved 2 days ago :) https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/

I'm pretty sure CIs other than GitHub's NIH had that figured out from the start already... That blog post doesn't tell how it's looking for free public organizations and repos and on github.com, maybe that's still broken?

Comment thread .github/workflows/main.yml Outdated
Comment thread .github/workflows/main.yml Outdated
@adriweb

adriweb commented Aug 5, 2020 via email

Copy link
Copy Markdown
Contributor

@adriweb

adriweb commented Oct 5, 2020

Copy link
Copy Markdown
Contributor

Any update?

I've asked @alberthdev to check if he can help finish this, too, it would be great to have this CI :)

@fridtjof fridtjof requested a review from Vogtinator October 6, 2020 00:17
@fridtjof

fridtjof commented Oct 6, 2020

Copy link
Copy Markdown
Contributor Author

Thanks for reminding me, @adriweb :)

Should be good now - CI times without the toolchain upload are down to 60-90 seconds. I'd say that is acceptable :)

When this is merged (and when I get around to it), I'm going to take a shot at macOS CI and put out another PR with that.

@fridtjof

fridtjof commented Oct 6, 2020

Copy link
Copy Markdown
Contributor Author

rebased on top of r2016 (also master as of now), CI in progress here: https://github.com/shinyblink/Ndless/actions/runs/290422251

@alberthdev

Copy link
Copy Markdown

Was asked to take a look at this - changes as they stand today look pretty good! You're also protected from this issue, of which I'll need to update sooner rather than later as well for another project...

Some comments addressing stuff mentioned above:

  • Security - it might help to filter on the event's activity types, at least from the perspective of only allowing activity types that a maintainer can actually trigger, e.g. labeling "ready-for-ci" or similar. See: pull_request event
  • Other platforms - if the resulting GitHub Actions is different enough that it can't really be easily done in one file, it's not a bad idea to make those distinct. Example would be if we're trying to build on Windows - that could maybe be done in this one (and only use WSL to build), or in a separate file with a build matrix toggling on or off the WSL build step. No strong opinion here though...
  • Older Linux - probably best to leave Docker out of the mix for now, if we ever add it maybe as a separate actions workflow file too. I'd (half-jokingly) suggest that if supporting older Linux distros is of interest, running this image could be fruitful: https://github.com/pypa/manylinux - and we could sort of rely on their judgement for maintenance. (Bonus if the resulting toolchain / binaries were outputted through actions/upload-artifact... then "many Linux users" would be able to run as well!)

@Vogtinator

Vogtinator commented Oct 6, 2020

Copy link
Copy Markdown
Contributor

It's looking good now! But before I merge, I have one question about GitHub actions: If someone makes a PR to master with a changed workflows/main.yml file, which one would the action run with which token with what permissions for which repo?

@Danke-Boi

Danke-Boi commented Oct 6, 2020 via email

Copy link
Copy Markdown

@Danke-Boi

Danke-Boi commented Oct 6, 2020 via email

Copy link
Copy Markdown

@lights0123

Copy link
Copy Markdown
Contributor

@Vogtinator no secret tokens are available in PRs (although they will be available in the creator's fork). If you merge them, however, they will become available.

@Vogtinator Vogtinator merged commit ef67a05 into ndless-nspire:master Oct 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants