Skip to content

Build native MacOS arm64 artifacts (universal2)#31747

Merged
gnossen merged 5 commits intogrpc:masterfrom
gnossen:attempt_universal2_native
Dec 1, 2022
Merged

Build native MacOS arm64 artifacts (universal2)#31747
gnossen merged 5 commits intogrpc:masterfrom
gnossen:attempt_universal2_native

Conversation

@gnossen
Copy link
Copy Markdown
Contributor

@gnossen gnossen commented Nov 22, 2022 •
edited
Loading

This PR builds all MacOS artifacts as universal2 artifacts by default. This builds the .whl with a shared object library supporting both the x86_64 and arm64 platforms:

rbellevi-macbookpro:tmp.u9QVKZQH rbellevi$ file ./grpc/_cython/cygrpc.cpython-311-darwin.so
./grpc/_cython/cygrpc.cpython-311-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64]
./grpc/_cython/cygrpc.cpython-311-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
./grpc/_cython/cygrpc.cpython-311-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64

This also removes the hack introduced by #29857 since we are now truly building universal artifacts. Fixes #29262

@gnossen gnossen added lang/Python release notes: yes Indicates if PR needs to be in release notes labels Nov 22, 2022
@gnossen gnossen changed the title Attempt to build universal2 artifacts Build universal2 artifacts Nov 23, 2022
@gnossen gnossen changed the title Build universal2 artifacts Build native MacOS arm64 artifacts Nov 29, 2022
@gnossen gnossen marked this pull request as ready for review November 29, 2022 00:34
@gnossen gnossen removed the request for review from XuanWang-Amos November 29, 2022 00:36
@gnossen gnossen marked this pull request as draft November 29, 2022 00:37
@gnossen
Copy link
Copy Markdown
Contributor Author

gnossen commented Nov 29, 2022 •
edited
Loading

Whoops. Looks like in the most recent build, only 3.11 artifacts ended up as universal. I guess this need some more work.

Ah. It looks like it's just the naming. All of the artifacts actually do have universal shared object libraries bundled:

rbellevi-macbookpro:tmp.sS60K8tS rbellevi$ file grpc_tools/_protoc_compiler.cpython-310-darwin.so
grpc_tools/_protoc_compiler.cpython-310-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64]
grpc_tools/_protoc_compiler.cpython-310-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
grpc_tools/_protoc_compiler.cpython-310-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64

@gnossen
Copy link
Copy Markdown
Contributor Author

gnossen commented Nov 29, 2022

Yet another issue:

(venv) rbellevi-macbookpro:tmp.u9QVKZQH rbellevi$ file  grpc/_cython/cygrpc.cpython-310-darwin.so
grpc/_cython/cygrpc.cpython-310-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64]
grpc/_cython/cygrpc.cpython-310-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
grpc/_cython/cygrpc.cpython-310-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64
(venv) rbellevi-macbookpro:tmp.u9QVKZQH rbellevi$ arch
arm64
(venv) rbellevi-macbookpro:tmp.u9QVKZQH rbellevi$ python
Python 3.10.7 (main, Nov 29 2022, 11:11:58) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import grpc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/var/folders/p1/18dr8yp97cs7mq__92r5fwfr00krr5/T/tmp.u9QVKZQH/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/private/var/folders/p1/18dr8yp97cs7mq__92r5fwfr00krr5/T/tmp.u9QVKZQH/grpc/_compression.py", line 20, in <module>
    from grpc._cython import cygrpc
ImportError: dlopen(/private/var/folders/p1/18dr8yp97cs7mq__92r5fwfr00krr5/T/tmp.u9QVKZQH/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (_ChaCha20_ctr32)

It looks like a symbol from boringssl is missing. It seems that this function is written in assembly and expected to be linked in separately. As far as I can tell, there is no arm64 implementation. We'll probably have to turn off boringssl assembly optimizations. Hopefully, we can do it just for the arm64 version and not the x86 version, but I'm not sure yet if that's possible.

@gnossen
Copy link
Copy Markdown
Contributor Author

gnossen commented Nov 30, 2022

It looks like selectively disabling ASM optimizations for just arm64 would be difficult. We'd likely have to compile to object files then link them together into a fat archive using (e.g.) lipo. For the moment, I think it's reasonable to just disable the assembly optimizations for our prebuilt MacOS artifacts. This will allow M1 users to start using things right away.

@gnossen gnossen marked this pull request as ready for review November 30, 2022 19:32
@gnossen gnossen changed the title Build native MacOS arm64 artifacts Build native MacOS arm64 artifacts (universal2) Nov 30, 2022
@gnossen
Copy link
Copy Markdown
Contributor Author

gnossen commented Dec 1, 2022

This should make it into 1.52.0, which is currently scheduled for release December 27th, but may slip by a week or so due to the holidays.

@gnossen gnossen merged commit 5933b52 into grpc:master Dec 1, 2022
@gnossen gnossen deleted the attempt_universal2_native branch December 1, 2022 17:57
@copybara-service copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Dec 1, 2022
@XuanWang-Amos XuanWang-Amos mentioned this pull request Dec 6, 2022
Closed
@gnossen gnossen mentioned this pull request Jan 10, 2023
Closed
@levrik levrik mentioned this pull request Jan 25, 2023
Closed
gnossen added a commit to gnossen/grpc that referenced this pull request Feb 17, 2023
@gnossen
Build native MacOS arm64 artifacts (universal2) (grpc#31747)
9b907d5 
* Attempt to build universal2 artifacts

* Whoops

* Reverse the hack

* Turn off boringssl assembly optimizations

* Whoopsie
gnossen added a commit that referenced this pull request Feb 17, 2023
@gnossen
Backport of #31747 to v1.51.x (Build native MacOS arm64 artifacts (un…
1141f04 
…iversal2)) (#32424)
@gibsondan gibsondan mentioned this pull request Feb 17, 2023
Open
gnossen added a commit to gnossen/grpc that referenced this pull request Feb 21, 2023
@gnossen
Build native MacOS arm64 artifacts (universal2) (grpc#31747)
64adfdb 
* Attempt to build universal2 artifacts

* Whoops

* Reverse the hack

* Turn off boringssl assembly optimizations

* Whoopsie
ejona86 pushed a commit that referenced this pull request Feb 22, 2023
@gnossen
Backport of #31747 to v1.47.x (Build native MacOS arm64 artifacts (un…
a954fbb 
…iversal2)) (#32446)
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@gnossen @wanlin31
Build native MacOS arm64 artifacts (universal2) (#31747)
87cde4c 
* Attempt to build universal2 artifacts

* Whoops

* Reverse the hack

* Turn off boringssl assembly optimizations

* Whoopsie
davidben added a commit to davidben/grpc that referenced this pull request Jul 13, 2023
@davidben
Simplify BoringSSL assembly build
b937ed8 
BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.

Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.

As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:

- grpc#31747 had to disable the assembly,
  because at the time assembly required the library be built differently
  for each architecture and then stitched back together. This should now
  work.

- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
  with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
  moot.
davidben added a commit to davidben/grpc that referenced this pull request Jul 13, 2023
@davidben
Simplify BoringSSL assembly build
64b1173 
BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.

Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.

As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:

- grpc#31747 had to disable the assembly,
  because at the time assembly required the library be built differently
  for each architecture and then stitched back together. This should now
  work.

- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
  with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
  moot.
@davidben davidben mentioned this pull request Jul 13, 2023
Merged
jtattermusch pushed a commit that referenced this pull request Aug 2, 2023
@davidben
[boringssl] Simplify BoringSSL assembly build (#33700)
113b092 
~~NB: I haven't tested this at all and am hoping the CI will tell me
where I've (undoubtedly) messed something up.~~ Edit: looks like CI is
now clear!

BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.

Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.

As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:

- #31747 had to disable the assembly,
because at the time assembly required the library be built differently
for each architecture and then stitched back together. This should now
work.

- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
moot.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@XuanWang-Amos XuanWang-Amos XuanWang-Amos approved these changes

Assignees

@XuanWang-Amos XuanWang-Amos

Labels

bloat/none imported Specifies if the PR has been imported to the internal repository lang/Python per-call-memory/neutral per-channel-memory/neutral release notes: yes Indicates if PR needs to be in release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

A solution to build Apple Silicon (M1) Python binary wheels

2 participants

@gnossen @XuanWang-Amos