Skip to content

Conversation

@hoffie
Copy link
Member

@hoffie hoffie commented Mar 9, 2022

Short description of changes

  • Autobuild: Merge Mac build scripts

    • diff -u autobuild/mac/{artifacts,codeQL}/autobuild_mac_1_prepare.sh is empty.
    • diff -u autobuild/mac/{artifacts,codeQL}/autobuild_mac_2_build.sh shows a different build process for artifacts and CodeQL. As we are combining both jobs now, taking just the artifacts build.sh is fine though.
    • autobuild/mac/artifacts/autobuild_mac_3_copy_files.sh does not exist for the CodeQL job (it's unused there).

    Therefore, it's safe to move those scripts one level up and drop all the duplication. That makes it consistent with all the other platforms as well.
    This commit moves the artifacts/*.sh files to their new location and adjusts a single line to account for the reduced nesting level. It also updates the callers to use the new paths.

  • Autobuild: Move ios build scripts up one level
    There's no need to place the ios autobuild scripts in an "artifacts" sub directory. This was probably done as the ios build was likely based on the mac build logic.
    This commit doesn't change any .sh file content, it just moves the files one level up and updates the callers to use the new paths.

  • Autobuild: Drop extra Mac CodeQL step
    All other platforms run CodeQL as part of their regular build. This increases the time until artifacts are available, but avoids having to build each job twice. We should handle codeQL consistenly.

CHANGELOG: Autobuild: Reorganized macOS/iOS build logic.

Context: Fixes an issue?

Fixes unneeded duplication.

Does this change need documentation? What needs to be documented and how?

No.

Status of this Pull Request

Ready.

What is missing until this pull request can be merged?

Reviews.

Checklist

  • I've verified that this Pull Request follows the general code principles
  • I tested my code and it does what I want
  • My code follows the style guide
  • I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
  • I've filled all the content above

@hoffie hoffie added this to the Release 3.9.0 milestone Mar 9, 2022
@pljones
Copy link
Collaborator

pljones commented Mar 9, 2022

All other platforms run codeQL as part of their regular build. This increases the time until artifacts are available, but avoids having to build each job twice. We should handle codeQL consistenly.

There was a reason for that, now I see it written out. Hm. It'll be explained somewhere - I think it was an answer to one of the points I raised.

@softins
Copy link
Member

softins commented Mar 9, 2022

  • diff -u autobuild/mac/{artifacts,codeQL}/autobuild_mac_2_build.sh is empty.

Really? On latest master for me, they differ:

tony@pi:~/jamulus/autobuild/mac $ diff *
diff artifacts/autobuild_mac_2_build.sh codeQL/autobuild_mac_2_build.sh
1c1
< #!/bin/sh -e
---
> #!/bin/bash
10,14d9
< SIGN=$1
< if [ -n "${SIGN}" ]; then
<     shift
< fi
<
23d17
< echo "Run deploy script..."
25,50c19,26
< # If we have certificate details, then prepare the signing
< if [[ "${SIGN}" != "sign_if_possible" ||
<         -z "${MACOS_CERTIFICATE_PWD}" ||
<         -z "${MACOS_CERTIFICATE}" ||
<         -z "${MACOS_CERTIFICATE_ID}" ||
<         -z "${NOTARIZATION_PASSWORD}" ||
<         -z "${KEYCHAIN_PASSWORD}" ]]
< then
<     sh "${THIS_JAMULUS_PROJECT_PATH}"/mac/deploy_mac.sh
< else
<     echo "Setting up signing, as all credentials found"
<
<     # Get the cert to a file
<     echo ${MACOS_CERTIFICATE} | base64 --decode > certificate.p12
<
<     # Set up a keychain for the build
<     security create-keychain -p "${KEYCHAIN_PASSWORD}" build.keychain
<     security default-keychain -s build.keychain
<     security unlock-keychain -p "${KEYCHAIN_PASSWORD}" build.keychain
<     security import certificate.p12 -k build.keychain -P "${MACOS_CERTIFICATE_PWD}" -T /usr/bin/codesign
<     security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "${KEYCHAIN_PASSWORD}" build.keychain
<
<     sh "${THIS_JAMULUS_PROJECT_PATH}"/mac/deploy_mac.sh -s "${MACOS_CERTIFICATE_ID}"
<     # Set up the notarization and staple parts
<     echo "::set-output name=macos_signed::true"
< fi
---
> echo "Building... qmake"
> qmake
>
> echo "Building... make"
> make -j "$(sysctl -n hw.ncpu)"
>
>
> echo "Done"
Only in artifacts: autobuild_mac_3_copy_files.sh

I haven't yet looked at your changes...

@hoffie
Copy link
Member Author

hoffie commented Mar 9, 2022

All other platforms run codeQL as part of their regular build. This increases the time until artifacts are available, but avoids having to build each job twice. We should handle codeQL consistenly.

There was a reason for that, now I see it written out. Hm. It'll be explained somewhere - I think it was an answer to one of the points I raised.

I've found some history regarding the Mac CodeQL split: #978 (comment) #978 (comment)

BTW, the initial cause for splitting artifacts and codeql in some cases was, that for some unkonown reason the build of the installers did not trigger the codeql-inspection.

@softins Thanks for the hint, I'll re-check my history. That shouldn't happen. :(
Edit: Found the issue, I ran git diff instead of just diff. Will have to re-work this PR.

@hoffie hoffie marked this pull request as draft March 9, 2022 17:58
@softins
Copy link
Member

softins commented Mar 9, 2022

I made a comment recently somewhere about why we had separate CodeQL and Artifacts steps for Mac, but I can't find it now. It's not here, and probably buried in one of the relevant PRs or issues. I said that I had tried to combine CodeQL and Artifacts for Mac without success last year, by setting uses_codeql: true on the artifacts job. I found my branch, from May 2021, and can see in my Actions tab that the workflow failed consistently at the time. I have just re-run the same workflow on the same commit (4a736b0), and this time it succeeded! The log for the old failed run is no longer available, but I suspect that there has been some improvement to the run environment or to CodeQL itself since my first attempt.

So it looks like we are now safe to combine them as per this PR (using the build script from artifacts, as you have done, not from CodeQL).

The only other thing I wasn't sure of is that the Mac build script actually builds twice - once for client and once for server. I don't know how that affects the CodeQL part.

@ann0see
Copy link
Member

ann0see commented Mar 9, 2022

once for client and once for server.

I think it's the same for Debian (but it's gui and headless) there.

I do also think that we should track as issue that alternatives should be considered (in terms of not building basically the same binary twice). JamulusServer.app should just run Jamulus.app with the -s argument.

@hoffie
Copy link
Member Author

hoffie commented Mar 9, 2022

once for client and once for server.

I think it's the same for Debian (but it's gui and headless) there.

Yep, thought the same.

I do also think that we should track as issue that alternatives should be considered (in terms of not building basically the same binary twice). JamulusServer.app should just run Jamulus.app with the -s argument.

Running Jamulus.app with -s might be possible (I haven't investigated where this "starter" is defined). Right now it is being implemented with a define (SERVER_BUNDLE) which changes the behavior of the actual binary.
I think both this case and Linux (HEADLESS) is problematic for building in a single go. I don't think the build system can detect differences in defines, so we're required to run a clean build.
But yeah, if you could open an Issue to track this, this would be good.

@hoffie
Copy link
Member Author

hoffie commented Mar 9, 2022

@softins Thanks for your insights, tests and explanations!

I've just checked the diff again. Although I missed the differences initially, I think they are fine (especially now that we are no longer running a different job for CodeQL). I'm therefore removing the Draft status without any further changes. I've edited the PR description.

@hoffie hoffie marked this pull request as ready for review March 9, 2022 23:36
@ann0see ann0see self-requested a review March 10, 2022 19:55
Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it builds, I'll approve it.

@ann0see ann0see requested a review from softins March 11, 2022 20:22
Copy link
Member

@softins softins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok. There is a pending merge conflict flagged by Github, but that appears to be just to do with indentation in the matrix section. So I'll approve it in advance of that being fixed ready to merge.

hoffie added 3 commits March 12, 2022 13:16
- `diff -u autobuild/mac/{artifacts,codeQL}/autobuild_mac_1_prepare.sh` is
  empty.

- `diff -u autobuild/mac/{artifacts,codeQL}/autobuild_mac_2_build.sh`
  contains extra logic for signing, but as this is controlled by a
  parameter, it is fine to merge anyway.

- `autobuild/mac/artifacts/autobuild_mac_3_copy_files.sh` does not exist
  for the CodeQL job (it's unused there).

Therefore, it's safe to move those scripts one level up and drop all the
duplication. That makes it consistent with all the other platforms as
well.

This commit moves the .sh files to their new location and adjusts a
single line to account for the reduced nesting level. It also updates
the callers to use the new paths.
There's no need to place the ios autobuild scripts in an "artifacts" sub
directory. This was probably done as the ios build was likely based on
the mac build logic.

This commit moves the .sh files to their new location and adjusts a single
line to account for the reduced nesting level. It also updates the callers
to use the new paths.
All other platforms run codeQL as part of their regular build. This
increases the time until artifacts are available, but avoids having to
build each job twice.
We should handle codeQL consistenly.
@hoffie hoffie force-pushed the merge-mac-autobuild-scripts branch from 22cac32 to 729bbd8 Compare March 12, 2022 12:22
@hoffie
Copy link
Member Author

hoffie commented Mar 12, 2022

Rebased, confirmed that all targets are green and that MacOS Non-legacy properly runs Code-QL. Merging.

@hoffie hoffie merged commit b039cb8 into jamulussoftware:master Mar 12, 2022
@hoffie hoffie deleted the merge-mac-autobuild-scripts branch March 19, 2022 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants