Skip to content

Sign Windows .exes in a post-build hook#2160

Merged
vilmibm merged 6 commits intocli:trunkfrom
mbpreble:sign-windows-executables
Jan 18, 2022
Merged

Sign Windows .exes in a post-build hook#2160
vilmibm merged 6 commits intocli:trunkfrom
mbpreble:sign-windows-executables

Conversation

@mbpreble
Copy link

@mbpreble mbpreble commented Oct 11, 2020

Fixes #2111

I've tested that the hook at least completes successfully (following tagging of a release) in my fork. This has been done using my own self-signed certificate in .pfx format, pulled down from my working branch via the Github API.

Signing is accomplished by Mono's signcode utility which is only shipped with sha256 support in ubuntu-20.04 (hence a bump here, but this could probably still run on ubuntu-latest if we manually get a newer version of Mono).

Mechanics of signing are intended to be more or less the same as that used for the MSI, there is some overlap here between the existing Windows-targeted scripts for signing the MSI and the new hook intended to sign the .exes on Linux.

Copy link
Contributor

@mislav mislav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Thank you so much for setting this up; it likely saves us a lot of time 🙇

One thing that I find sub-optimal about our whole build process now is that we sign .exe on Linux and the .msi on Windows, i.e. that the signing logic is basically spread across different build strategies. I'm not familiar with Mono; do you know if it's possible to also build the .msi installer on Linux under Mono? Currently we use the Wix toolset on Windows to build the installer, but this is the last bit of the build that we actually need to spin up a Windows job for.

jobs:
goreleaser:
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be fine to leave this as ubuntu-latest?

Copy link
Author

@mbpreble mbpreble Oct 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ubuntu-20.04 accomplishes an upgrade from mono 4.6 to 6.8.
It's needed here to enable use of SHA256 signing in signcode (-a sha256) (see mono/mono#7517)

A couple of alternatives:

@mbpreble
Copy link
Author

This looks great! Thank you so much for setting this up; it likely saves us a lot of time

One thing that I find sub-optimal about our whole build process now is that we sign .exe on Linux and the .msi on Windows, i.e. that the signing logic is basically spread across different build strategies. I'm not familiar with Mono; do you know if it's possible to also build the .msi installer on Linux under Mono? Currently we use the Wix toolset on Windows to build the installer, but this is the last bit of the build that we actually need to spin up a Windows job for.

There doesn't appear to be (at least i couldn't find it) a utility for this shipped with Mono.
The most obvious thing for MSIs that I could find is msitools (https://wiki.gnome.org/msitools) which should be available as a package for Ubuntu (probably have to install it during the GitHub action). That might be a path to replacing the Windows-specific MSI building + signing code with a utility that runs on Linux and the signing script provided here.

Copy link

@boss562 boss562 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linux repositories

@vilmibm vilmibm assigned vilmibm and unassigned mislav Jun 2, 2021
Copy link

@7flores11 7flores11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mislav mislav added the external pull request originating outside of the CLI core team label Dec 20, 2021
@mbpreble mbpreble requested a review from a team as a code owner January 6, 2022 18:35
@mbpreble mbpreble requested review from samcoe and removed request for a team January 6, 2022 18:35
@vilmibm vilmibm self-requested a review January 6, 2022 18:36
@vilmibm
Copy link
Contributor

vilmibm commented Jan 7, 2022

It's late to an embarrassing degree but I've finally grabbed a binary and tested it. It works fine, and I can see that goreleaser ran the sign step, but the .exe does not appear to actually be signed:

image

I'll poke around at the script and see if I can determine where it's failing

@vilmibm vilmibm force-pushed the sign-windows-executables branch from d5d0e31 to f98e7e4 Compare January 7, 2022 18:31
@mbpreble
Copy link
Author

mbpreble commented Jan 7, 2022

I believe I fell short of verifying that the output files were actually signed when originally working through this - getting the action itself to complete seems to be where I kind of dropped off. That was in a heavily modified working branch to boot, running as little as the actual release machinery as I could get away with - that didn't include important things like actually saving the .exes anywhere outside of the build process.

Will definitely stare at the thing to see if anything jumps out at me but any testing of this in an actual branch of the actual repo by tagging releases (looks like that's what we're doing) is hopefully going to be easier to debug than whatever I'd need to be doing.

@mbpreble
Copy link
Author

mbpreble commented Jan 7, 2022

We seem to be looking happy for 386 but not amd64? (files in the v2.5.0-pre5 release)

386:

chktrust -v gh.exe
Mono CheckTrust - version 4.6.2.0
Verify if an PE executable has a valid Authenticode(tm) signature
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Verifying file gh.exe for Authenticode(tm) signatures...

INFO! gh.exe was timestamped on 1/7/2022 7:14:56 PM
SUCCESS: gh.exe signature is valid
and can be traced back to a trusted root!

amd64:

chktrust -v gh.exe
Mono CheckTrust - version 4.6.2.0
Verify if an PE executable has a valid Authenticode(tm) signature
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Verifying file gh.exe for Authenticode(tm) signatures...

WARNING! gh.exe is not timestamped!
ERROR! gh.exe doesn't contain a digital signature!

Edit: perhaps I shouldn’t expect chktrust to work with a 64 bit executable at all, though. Its docs don’t claim that it can do this. The docs for signcode don’t claim to be able to sign 64 bit executables either.

I suspect we’ve gotten this to work for 32 bit but not so much for 64.

@mbpreble
Copy link
Author

mbpreble commented Jan 7, 2022

I can probably take a run at getting this moved over to osslsigncode to make it work for 64 bit.

The fine folks at Mozilla have been on a similar journey:
https://bugzilla.mozilla.org/show_bug.cgi?id=711210

@vilmibm
Copy link
Contributor

vilmibm commented Jan 7, 2022

hello @mbpreble ! thanks for jumping back into this PR after we let it lie for way too long 😓 I didn't even think to check the 32bit binary, interesting discovery.

Getting it over to osslsigncode sounds awesome; let me know how I can help (including doing runs and grabbing binaries to run on my windows machine as needed).

@vilmibm
Copy link
Contributor

vilmibm commented Jan 7, 2022

oh, for thoroughness I should also mention that @mislav is working on getting us off of the desktop secrets's cert; so we will need to switch to a different signing cert soon. Mislav, do you know what the approach is to getting that cert in place for a script like this?

@mbpreble
Copy link
Author

mbpreble commented Jan 8, 2022

@vilmibm - Think I've got this moved over to osslsigncode, have copied the changes from my testing branch into a new commit. When I run a release in my fork it seems to successfully sign both the 386 and amd64 binaries - I ran osslsigncode verify against them and they appear to be signed by my self-signed cert.

Would recommend trying this as you've been doing in the context of a branch within the actual repo, adding a tag, and kicking the tires on the executables.

@vilmibm
Copy link
Contributor

vilmibm commented Jan 10, 2022

hm, it looks like the 386 signing went fine in my prerelease build, but the 64bit one failed:

the logs are a bit garbled, but:

Failed to read certificate file: certificate.pem\n140472794503808:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:../crypto/asn1/asn1_lib.c:101:\n140472794503808:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:\n140472794503808:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:\n\nFailed\n": exit status 255
goreleaser      Run GoReleaser  2022-01-10T20:57:40.2552336Z ##[error]The process '/opt/hostedtoolcache/goreleaser-action/0.174.1/x64/goreleaser' failed with exit code 1

@vilmibm
Copy link
Contributor

vilmibm commented Jan 10, 2022

I'm actually wondering if the curl is failing the second time, somehow. I'm going to do some debugging around that.

@vilmibm
Copy link
Contributor

vilmibm commented Jan 10, 2022

hm, now I'm not sure. the hooks run non-deterministically, so I can't actually confirm that one of them is working. I did get a different error the second time I did the prerelease build -- the gh_signed.exe binary was not found by mv (but no error made it to the logs prior to the mv). I'll try and get some debugging in there and do some more builds.

@vilmibm
Copy link
Contributor

vilmibm commented Jan 11, 2022

I'm really at a loss, here. the final mv invocation fails because it can't stat the exe. it clearly exists prior to the sleep. hopefully it'll make some sense in the morning.

@vilmibm
Copy link
Contributor

vilmibm commented Jan 11, 2022

@mislav agrees that it seems like hooks running in parallel are the culprit, here. I'm going to de-generalize this into a single hook.

@vilmibm
Copy link
Contributor

vilmibm commented Jan 12, 2022

I factored out code that would clobber itself when run in parallel into a single global before hook. Now, the signing appears to succeed without issue.

I'm out and about on my mac book but later today I'll grab the .exe files and make sure everything looks good. I'll also clean up the git history ^_^()

@mbpreble
Copy link
Author

You're doing a hero's work on this one. You might be surprised to know I had no previous experience with GoReleaser and had no idea the hooks would run in parallel, in the same workspace. 🙃

@vilmibm
Copy link
Contributor

vilmibm commented Jan 13, 2022

Looks good to go!

image

I'm going to do a force push to clean up my nonsense commits then merge this in.

@vilmibm vilmibm force-pushed the sign-windows-executables branch from 9336cec to f30b7db Compare January 13, 2022 19:38
@vilmibm
Copy link
Contributor

vilmibm commented Jan 13, 2022

Leaving this for either @mislav or @samcoe to approve given how many commits I ended up adding.

Copy link
Contributor

@samcoe samcoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all seems good to me. Left one non-blocking question.

.goreleaser.yml Outdated
hooks:
- go mod tidy
- make manpages GH_VERSION={{.Version}}
- ./script/prepare-windows-cert.sh "{{.Env.GITHUB_CERT_PASSWORD}}" "{{.Env.DESKTOP_CERT_TOKEN}}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason not to do this in the windows build section in a pre hook?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is!

the build matrix for windows (386 and amd64) means that the hook would be run twice in parallel. the curl would run twice and there are a pile of race conditions that occur as a result.

Copy link
Contributor

@mislav mislav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! However, this set of changes prevents me from running goreleaser locally to test out releases.

⨯ release failed after 3.19s error=template: tmpl:1:40: executing "tmpl" at <.Env.GITHUB_CERT_PASSWORD>: map has no entry for key "GITHUB_CERT_PASSWORD"

Could we make it so that Windows signing is skipped if the information necessary to fetch the certificates isn't present in the environment?

@vilmibm
Copy link
Contributor

vilmibm commented Jan 14, 2022

@mislav I've:

  • added newlines to end of files
  • made local publishing work
  • removed my stray debugging stuff

@vilmibm vilmibm requested a review from mislav January 14, 2022 22:58
Copy link
Contributor

@mislav mislav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!! 🚀

@vilmibm vilmibm merged commit 8c862bb into cli:trunk Jan 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external pull request originating outside of the CLI core team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sign gh.exe binaries in addition to signing the MSI installer

6 participants