Skip to content

Improve CI#2328

Closed
tcharding wants to merge 19 commits intorust-bitcoin:masterfrom
tcharding:01-10-ci
Closed

Improve CI#2328
tcharding wants to merge 19 commits intorust-bitcoin:masterfrom
tcharding:01-10-ci

Conversation

@tcharding
Copy link
Copy Markdown
Member

@tcharding tcharding commented Jan 10, 2024

This is now an epic re-work of the CI scripts. The aims are:

  • Make it obvious what shell script code is running in which github actions job.
  • Make the actions quicker to run and quicker to fail.
  • Make it obvious when a CI job is red what failed without having to scroll through so much log file noise.
  • Make it more clear what is happening to the lock file when the CI script runs.
  • Reduce duplicate shell code.
  • Make the whole CI setup more maintainable.
  • You get the picture :)

I've made the changes really small and said why in each commit log - I did this, more than usual, so as to help catch either bugs in the change set or bugs that currently exist in the CI scripts but we didn't notice them.

@apoelstra
Copy link
Copy Markdown
Member

It's hard to tell where you're going with the first commit. Will CRATES and DEPS become arguments to these functions?

The second commit looks good to me.

@tcharding
Copy link
Copy Markdown
Member Author

Oh I just split the first commit out because I thought there might be some comments on the style. We haven't used this style of shell in this project yet (I think), putting everything in functions and calling main from the bottom of the script. I learned this style from the old rustup shell script back in the day IIRC.

@apoelstra
Copy link
Copy Markdown
Member

Ah! If it's purely stylistic then concept ACK.

@tcharding
Copy link
Copy Markdown
Member Author

Cool I'll flesh this PR out then. Thanks man.

@tcharding tcharding force-pushed the 01-10-ci branch 4 times, most recently from 574bbb5 to f80b57e Compare January 11, 2024 05:52
@tcharding
Copy link
Copy Markdown
Member Author

The lint fail is due to the duplicate deps test, and locally I cannot reproduce?

@tcharding
Copy link
Copy Markdown
Member Author

And the ASAN stuff is ugly.

@tcharding tcharding marked this pull request as ready for review January 11, 2024 05:59
@tcharding
Copy link
Copy Markdown
Member Author

shellchecked the whole lot!

Add a `main` function to script in preparation for adding more
functions. Move the current script into a function and call it from
main.

Refactor only, no logic changes.
Use `bash` in all the CI `test.sh` scripts, for no particular reason
other than to be uniform with other scripts in the repository.
Instead of running the linter in each individual CI script we can just
run it once from the top level script and exit. This has the benefit of
being quicker with no loss of linting coverage. Note this does remove
sanity builds from the linting job, this does not reduce overall
coverage of the whole CI workflow.
As we did for the linting stuff, move the docs stuff out of each
individual CI script and into the top level script. Reducing code
duplication with no reduction in test coverage.
These MSRV pinning blocks are stale, remove them.
The CI script is overly complex to reason about because of how it
interacts with the github actions. In an effort to make it more obvious
exactly what is going on and what is failing when CI is red split the
linting and docs stuff out into separate jobs. In the CI script once
these are run exit. This has the added advantage of making these jobs
fast to run and fast to fail, hopefully making the dev feedback loop
quicker.
We do not run the formatter in CI anymore, remove the stale logic from
all test scripts.
@tcharding tcharding force-pushed the 01-10-ci branch 2 times, most recently from 3a43af8 to 369f6bf Compare January 14, 2024 19:41
We can export this from the top level script, thereby reducing code
duplication. If devs want this when running individual test scripts they
can set it in the script invocation.
As we did for linting and docs; move the bench stuff to the top level
script and run it in a separate job.
These variables are no longer used, remove them.
The `AS_DEPENDENCY` test does not need to be run for the recent and
minimal lock files, just run it once in a separate job.
This test does not use the top level lock file, no need to run it
multiple times, just run it once from the top level script and do so
from a separate CI job.
The ASAN tests do not use `cargo --locked` so can be run from the top
level CI script.

This patch _increases_ coverage because the job that enables DO_ASAN was
only being run in `hashes/` - now we run it in `fuzz/` as well.

Note please I'm not very knowledgeable about ASAN stuff or what exactly
is going on in these commands but I remove `--lib` (and added `+nightly`
to be explicit) in order to get the it working in `fuzz/`.
The WASM test does not use `cargo --locked` so we can run it from the
top level CI script.
The individual CI scripts are expected to be run after copying the
recent/minimal lock files into place. As such all `cargo` commands
should be run with `--locked` otherwise the lockfile will be updated. To
save us making mistakes in the future introduce a variable `CARGO` and
always use it instead of `cargo`.

This patch changes the current behaviour because in some places the
sanity checks are being run without using `--locked` - this is a bug.
The current CI github action uses the name 'Continuous integration'
which is really long and noisy - use CI with no loss of clarity.
TIL, one can use curly braces and only redirect a single time - thanks
`shellcheck`.

Refactor only, no logic changes.
`shellcheck` complains about unused `FEATURES` variable, leave it there
commented out to make explicit the non-uniformity of this script
compared to the others. I'd say there is some work left to do to clean
up the feature matrix stuff.
Add a separate github action to run the duplicate dependencies job.
@tcharding
Copy link
Copy Markdown
Member Author

A bit of help here please.

  • I cannot repro the duplicate dependencies job fail?
  • I cannot workout why syn is showing up when we grep -v for it?

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Jan 15, 2024

OMG, I've just started doing pretty much this exact thing. I'll defer to you.

I chose a bit different approach:

  • There's a contrib/test_vars.sh in each crate defining FEATURES LINT_EXAMPLES and RUN_EXAMPLES
  • I put the crate and DEPS list in a single file where they're read by the top-level contrib/test.sh script and by GH actions supplying it into the matrix (this is a bit involved, I can help with this, I've already done it in other project)
  • I've put crate-specific things (like schemars) into $crate/contrib/extra_tests.sh
  • I removed almost all env vars (only schemars stays) and turned them into an argument of a helper run_task script
  • I defined matrix of crate, task, deps for each Rust version

The advantages are: even more parallel builds - each crate and task gets its own. I've had plans to add caching as well, even more de-duplication of the script - it's practically completely de-duplicated since all logic is unified. Only exceptional cases are handled in separate script files.

@tcharding
Copy link
Copy Markdown
Member Author

Sweet, can you push up what ever you have done to a branch somewhere so I can grab it and put it into this PR?

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Jan 16, 2024

See #2343 (there's clearly something broken since Prepare job didn't run, who would have guessed? :D)

@tcharding tcharding mentioned this pull request Jan 18, 2024
@tcharding
Copy link
Copy Markdown
Member Author

Closing in favour of #2353

@tcharding tcharding closed this Jan 23, 2024
apoelstra added a commit that referenced this pull request Feb 2, 2024
01a66a7 CI: Check for required commands (Tobin C. Harding)
5c15ed5 CI: Epic overhaul (Martin Habovstiak)
242aa67 Use env bash instead of /bin/bash (Tobin C. Harding)
422d301 Use bash to run shell scripts (Tobin C. Harding)

Pull request description:

  The combination of some work by myself [0] and Kix [1].

  Draft so I can use github's infrastructure to test it all out.

  Includes some patches at the front to fix real issues that the new test infrastructure found - WIN.

  [0] #2328
  [1] #2343

  Coincidentally this closes 1124

  Resolve: #1124

ACKs for top commit:
  apoelstra:
    ACK 01a66a7

Tree-SHA512: 026a0948a181102246702eadc3ff245c319c456b03ada9ca269141d006146f30fd8eb50377062735a06c3e369f7edac2e334587120338a3747810d999177d930
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants