Moving to 2.1 from 2.0 can potentially mess up a user's keychain (as the caveats in 2.1 say). However this ends up working, I don't think it should auto-upgrade someone from 2.0 to 2.1.

To provide some clarity on the keyring issue when upgrading, from https://www.gnupg.org/faq/whats-new-in-2.1.html#nosecring:

Removal of the secret keyring
gpg used to keep the public key pairs in two files: pubring.gpg and secring.gpg. The only difference is that secring stored in addition to the public part also the private part of the key pair. The secret keyring thus contained only the keys for which a private key is available, that is the user’s key. It required a lot of code to keep both versions of the key in sync and led to sometimes surprising inconsistencies.

The design of GnuPG-2 demands that only the gpg-agent has control over the private parts of the keys and the actual encryption engine (gpg or gpgsm) does not know about the private key but care only about session keys and keys for symmetric encryption. This has been implemented about 10 years ago for gpgsm (the S/MIME part of GnuPG). However, gpg (the OpenPGP part) used the gpg-agent only as passphrase entry and cache device but handles the private key itself.

With GnuPG 2.1 this changed and gpg now also delegates all private key operations to the gpg-agent. Thus there is no more code in the gpg binary for handling private keys. En passant this allows the long time requested “merging of secret keys” and several other advanced key management techniques.

To ease the migration to the no-secring method, gpg detects the presence of a secring.gpg and converts the keys on-the-fly to the the key store of gpg-agent (this is the private-keys-v1.d directory below the GnuPG home directory (~/.gnupg)). This is done only once and an existing secring.gpg is then not anymore touched by gpg. This allows co-existence of older GnuPG versions with GnuPG 2.1. However, any change to the private keys using the new gpg will not show up when using pre-2.1 versions of GnuPG and vice versa.

Note that the command --export-secret-keys still creates an OpenPGP compliant file with the secret keys. This is achieved by asking gpg-agent to convert a key and return it in the OpenPGP protected format. The export operation requires that the passphrase for the key is entered so that gpg-agent is able to change the protection from its internal format to the OpenPGP required format.

EDIT:
Changes from the pubring.gpg format to pubring.kbx format, https://www.gnupg.org/faq/whats-new-in-2.1.html#keybox

The format GnuPG has always used for the public keyring is actually a slightly extended version of the on-the-wire format for OpenPGP key exchange. This format is quite inflexible to work with when random access to keys in the keyring is required. In fact gpg always parsed all keys in the keyring until it encountered the desired one. With a large keyring (more than a few thousand keys) this could be quite slow.

From its very beginning gpgsm has used a different format to store public keys (certificates) which we call a keybox. That file format carries meta information about the stored keys and thus allows searching without actually parsing the key and computing fingerprints and such. The keybox format has been designed to be protocol independent and with 2.1 support for OpenPGP keys has been added. Random access to the keys is now really fast and keyrings with 30000 keys and more are now easily possible. That change also enables us to easily introduce other storage methods

If no pubring.gpg is found, gpg defaults to the new keybox format and creates a pubring.kbx keybox file. If such a keybox file already exists, for example due to the use of gpgsm, it will also be used for OpenPGP keys. However, if a pubring.gpg is found and no keybox file with OpenPGP keys exists, the old pubring.gpg will be used. Take care: GnuPG versions before 2.1 will always use the pubring.gpg file and not know anything about keys stored in the keybox file.

To convert an existing pubring.gpg file to the keybox format, you first backup the ownertrust values, then rename the file to (for example) publickeys, so it won’t be recognized by any GnuPG version, then run import, and finally restore the ownertrust values:

$ cd ~/.gnupg
$ gpg --export-ownertrust >otrust.lst
$ mv pubring.gpg publickeys
$ gpg --import-options import-local-sigs --import publickeys
$ gpg --import-ownertrust otrust.lst

You may then rename the publickeys file back so that it can be used by older GnuPG versions. Remember that in this case you have two independent copies of the public keys. The ownertrust values are kept by all gpg versions in the file trustdb.gpg but the above precautions need to be taken to keep them over an import.

@chdiza as written, this will indeed upgrade gnupg2 people to 2.1.19.

@JCount I was thinking specifically about the new "keybox" format for the public keyring (described later on down in that webpage), not about the secret keyring. The former could really mess people up, especially if they're not expecting their normal brew updating/upgrading to make non-backwards-compatible keyring format changes.

this will indeed upgrade gnupg2 people to 2.1.19.

Well, that's bad. If I recall correctly, this keybox thing is pretty much the whole reason why 2.1 was kept out in versions for so long, and then not made the default gnupg. Nothing has changed, as far as I know. I defer to @DomT4 on this.

Nothing has changed, as far as I know.

2.0 will be EOL 31 Dec 2017.

Yes, @chdiza is correct. Whilst the move by default to 2.1.x will not destroy existing keys it will convert them to the new format and then keep them in the new format; IIRC certain changes made within the new format cannot be accessed by 1.x or 2.0.x ever.

So for example if you have the more portable gnupg1 installed alongside the newer gnupg2.1.x and you rely on the former being able to see what's going on with the latter things will break for you at some point, potentially immediately depending on how you use gpg.

The big sticking point for me on this was always that Homebrew does not intentionally mess with stuff in people's $HOME folders, and this almost violates that principle because of both Homebrew's automatic update system & the upgrade --all functionality baked into a vanilla brew upgrade.

It's not impossible, to say the least, that someone runs brew upgrade, ignores the significant wad of output because most people don't update wildly often, and then runs some gpg command without thinking about it and introduces a change gpg1 cannot see. No warning at all if you miss the upgrade messages.

Longer-term this also makes it necessary to rethink Homebrew's (incomplete because I'm a doofus) GPG verification, and Cask's (incomplete because nobody finished it) GPG verification if the plan there is to allow using existing keychains so people can still brew install stuff that needs/wants GPG verification offline, in the same way Homebrew technically supports you brew fetch-ing things in advance now and then brew install xyz whilst you're sat on a bus in the middle of nowhere.

@DomT4 Mind reviewing the PR assuming the upgrade is a foregone conclusion ;)

I'll look it over. Just wanted to lay out that this wouldn't be the end of the story if the update was pushed through, assumptions made elsewhere would need to change as well.

FWIW my long-term plan here prior to departing was to get to the point where gnupg2.1 didn't need patching or tests disabling to actually run a clean testsuite on macOS, then announce it somewhere a month or two in advance in both caveat form and a more explicit text document somewhere to reference, and then push it through after that notice period. That plan still has the issues of adjusting long-term assumptions, but my aim was less sudden change rather than less change 😄.

It would be nice if I could wave a magic wand and undo the years of anti-2.1 propaganda we've been putting out there.

Well, to be fair, there were some pretty gnarly issues towards the start of this branch, not least around the agent on upgrade. It was a legitimate decision to stop casual users going "Ooh new shiny thing, let's try it out!" without understanding the changes around that.

@chdiza I meant to post the part about the public keybox format but lost track of doing it. However, I think that its impacts on the end user are generally less in the event of an upgrade. This is because if there is an existing pubring.gpg file and no prior existing keybox, as would most likely be the case if you had been using the gnupg2 formula, as there is no automatic conversion of pubring.gpg. GnuPG would continue to use the the pubring.gpg file and remain backwards compatible.

It would only be an issue if prior use of gpgsm or something similar had generated a keybox. In this case, if the user were unaware it could potentially be a serious issue.

@DomT4 PR refreshed.

However, I think that its impacts on the end user are generally less in the event of an upgrade. This is because if there is an existing pubring.gpg file and no prior existing keybox, as would most likely be the case if you had been using the gnupg2 formula, as there is no automatic conversion of pubring.gpg. GnuPG would continue to use the the pubring.gpg file and remain backwards compatible.

Aye, this is correct, with some caveats. The biggest caveat is that gpg1 cannot automatically see new secret keys created by gpg2, so once you change your secret key (which theoretically we should all be doing a lot more than anyone actually does) gpg1 won't know about it unless you do the export/import dance.

This is only true for formats it actually understands though, if you use ECC (which IMO is half the point of the new branch) and then try to use that with gpg1 at all it does some weird weird things:

sec      0C/0xFFFFFFFFFFFFFFFF 2017-03-14
      Key fingerprint = 0000 0000 0000 0000 0000  0000 0000 0000 0000 0000
uid                            Shaun Sheep <shaunthesheep@gmail.com>
ssb      0e/0xFFFFFFFFFFFFFFFF 2017-03-14

So whilst it's true that the pain for existing users isn't immediately in your face, the longer you actually use the 2.1.x branch the more it becomes practically difficult to go back to 1.x or 2.0.x.

@DomT4 PR refreshed.

You fixed some things I was about to point out. That's very efficient 😄.

Filippo is correct on this point FWIW: #11080 (comment). I don't have an especially elaborately-modified GPG configuration, but I still get yelled at every time I run a command under the new gpg.

Maybe that's not a bad thing, means people might actually read up and adjust their configurations, but it will require user intervention and user knowledge of what needs to change and why, which could be interesting.

It may require some hand holding but I'm sure upstream will help pick up some of the support burden.

@DomT4 Thank you for your constructive explanation of how I was underestimating the potential for issues that the end user faces. I appreciate candor, especially when it comes to things I may have misinterpreted or misunderstood. 😄

This looks right to me, in general.

Since this doesn't seem to have been mentioned in the thread, I'll point it out: GPG Suite, a well established GnuPG distribution on macOS that provides Mail.app integration and so on, currently uses GPG 2.0.30. I'm afraid that upgrading to 2.1.x (which automatically upgrades the keyring format in place) would destroy interoperability with GPG Suite.

I think a caveat should be added to warn users to uninstall brewed gnupg 2.1.x if they use GPG Suite.

I think a caveat should be added to warn users to uninstall brewed gnupg 2.1.x if they use GPG Suite.

That's not a particularly constructive suggestion.

That's not a particularly constructive suggestion.

What would you suggest? Data loss (okay, maybe not loss, but close) is a serious issue.

I would suggest you contact GPG Suite upstream and recommend that they share their plan for 2.0 EOL with everyone.

All I know about "2.0 EOL" is from https://www.gnupg.org/download/index.html:

GnuPG modern (2.1) is the brand new version with enhanced features like support for Elliptic Curve Cryptography. It will eventually replace the current stable (2.0).

"Eventually" can mean anything, hence nothing. Have you heard of a more concrete plan? Or are you suggesting that I torture a plan out of the GPG team first?

Nevertheless, I'm not sure how that suggestion conflicts with my practical suggestion of not having both brewed gpg 2.1.x (assuming it will land soon) and GPG Suite installed at the same time. You simply can't use both without messing up, unless you have separate GNUPGHOMEs, which is unlikely.

Also note that GPG Suite can be installed via brew-cask (brew cask install gpgtools) so you can't tie up your hands and say "none of our business".

https://www.gnupg.org/

"2.0.30 is the stable version from an often used branch. This branch will reach end-of-life on 2017-12-31."

Okay, created support ticket https://gpgtools.tenderapp.com/discussions/problems/51244. (Code is at https://github.com/GPGTools/MacGPG2, but issue tracker isn't enabled.)

In the meantime, "my practical suggestion of not having both brewed gpg 2.1.x (assuming it will land soon) and GPG Suite installed at the same time" still stands.

@DomT4 Also: Homebrew/brew#2359.

I vaguely recall someone on the team (can't remember who though so maybe not) had intentions to leave migrations permanently available via the old name, but I'll leave that down to y'all to resolve 🙈.

Technically that would still be the case here, since gnupg@1.4 and gnupg@2.0 are new formulae, not migrations from anywhere.

@ilovezfs Just a heads-up that I fly out Wednesday and won't be taking my laptop with me, so I'll likely be off-the-grid or confined to short emailed replies till early April. This may be a relief for you .

The new tag has shipped, and we're back to 🍏 here.

Shipped. The die is cast!

Thanks to everyone who helped here, particularly @JCount, @DomT4, and @MikeMcQuaid.