This should now be possible, see comment upstream. This would also collapse the builds into one per aarch (meaning we have to either exempt the aarch/ppc builds from this, or figure out cross-compiling for CUDA).