BLD: use macos-11 image on azure, macos-1015 is deprecated by mattip · Pull Request #22043 · numpy/numpy

mattip · 2022-07-25T12:16:09Z

There is a warning in the CI runs that the macos-1015 image is deprecated.

rgommers · 2022-07-25T17:16:40Z

Good idea to update. Some of the f2py tests do not seem happy.

The pytest output is pretty terrible, so it's not clear from the summary which 6 tests are failing - I only see a long list of XFAIL and SKIPPED.

mattip · 2022-07-25T19:08:50Z

The error is a failure to compile. Do the tests work locally on macOS 11?

ld: library not found for -lSystem

rgommers · 2022-07-25T19:32:04Z

The error is a failure to compile. Do the tests work locally on macOS 11?

Yes, but macOS 11 isn't really a thing many people use; it was followed by 12.0 shortly after, and we're at 12.4 now.

There is gh-20677 which looks like the same problem. However, it's kind of weird in this PR that the regular build works but the test suite fails on compiling something. So perhaps it has to do with the way the f2py tests are run, since a linker flag goes missing.

mattip · 2022-08-11T07:09:33Z

What would be the downside of moving to a macos-12-based image? Does that mean the wheels cannot be used on macos < 12?

rgommers · 2022-08-11T10:53:50Z

For arm64 wheels it is fine to require macOS >= 12.0, because 11.0 was quite buggy and short-lived. A number of fixes that we needed in numpy for scipy ended up in macOS 12.1 and the relevant Clang versions. SciPy already requires >= 12.0; I can't imagine that there are many arm64 users on 11 still.

mattip · 2022-08-15T11:42:52Z

Even with macOS-12 the error of ld: library not found for -lSystem is still there. Here is the compiler command:

INFO: Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
E                                  Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
E                                  Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
E                                  INFO: compile options: '-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/src.macosx-10.15-x86_64-3.8 -I/Users/runner/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/numpy-1.24.0.dev0+638.g596fb0dcf-py3.8-macosx-10.15-x86_64.egg/numpy/core/include -I/Users/runner/hostedtoolcache/Python/3.8.13/x64/include/python3.8 -c'
E                                  extra options: '-J/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/ -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/'
E                                  INFO: gfortran:f90: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/src.macosx-10.15-x86_64-3.8/_test_abstract_interface_TestAbstractInterface_ext_module-f2pywrappers2.f90
E                                  INFO: /usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/src.macosx-10.15-x86_64-3.8/_test_abstract_interface_TestAbstractInterface_ext_modulemodule.o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/src.macosx-10.15-x86_64-3.8/fortranobject.o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpg01ush4w/foo.o /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp3ofky4jb/src.macosx-10.15-x86_64-3.8/_test_abstract_interface_TestAbstractInterface_ext_module-f2pywrappers2.o -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin13/4.9.0 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin13/4.9.0/../../.. -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin13/4.9.0/../../.. -L/Users/runner/hostedtoolcache/Python/3.8.13/x64/lib -lgfortran -o ./_test_abstract_interface_TestAbstractInterface_ext_module.cpython-38-darwin.so
E                                  ld: library not found for -lSystem

rgommers · 2022-08-15T16:42:58Z

I think you can go back to macos-11 then. We are doing something fishy somewhere to make -lSystem appear; I can find very few other references for this error.

This looks suspicious but is no longer used, so doesn't seem like the problem:

[ -n "$USE_XCODE_10" ] && /bin/bash -c "sudo xcode-select -s /Applications/Xcode_10.app/Contents/Developer"

Somehow -lSystem is only injected for gfortraninvocations. The code ingnu.pyappends flags fromsysconfig.get_config_vars(), how about adding some debug prints to see if in CI any of the entries returned from there include System`?

mattip · 2022-08-16T11:36:14Z

On a mac, I see -lSystem in CONFIG_ARGS (not relevant), LIBS and SHLIBS. But I don't see LIBS nor SHLIBS used in NumPy. I did find this stack overflow about -lSystem and gfortran, and another that suggest some interplay between gcc, gfortran, and xcode.

sysconfig variables

% python3 -m sysconfig |grep "lSystem" 
	CONFIG_ARGS = "'-C' '--host=x86_64-apple-darwin' '--build=x86_64-apple-darwin' '--enable-framework=/Applications/Xcode.app/Contents/Developer/Library/Frameworks' '--with-framework-name=Python3' '--with-openssl=/AppleInternal/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.Internal.sdk/usr/local/libressl-2.8' '--with-openssl-ldflags=-L/AppleInternal/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.Internal.sdk/usr/local/libressl-2.8/lib -F/AppleInternal/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.Internal.sdk/System/Library/PrivateFrameworks' '--with-openssl-libs=-framework TrustEvaluationAgent' '--with-bundleid-prefix=com.apple' '--enable-ipv6' '--prefix=/Applications/Xcode.app/Contents/Developer/usr' '--with-pymalloc' '--with-cxx-main=xcrun -sdk macosx clang -arch arm64 -arch x86_64' 'PYTHON_FOR_BUILD=PYTHONPATH=/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-110/Python/Lib DYLD_FRAMEWORK_PATH=/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects/host /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects/host/python.exe' 'TRAIN_STYLE=DT' 'CC=xcrun -sdk macosx clang' 'CXX=xcrun -sdk macosx clang++' 'CPP=xcrun -sdk macosx clang -E' 'CFLAGS=-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration' 'CPPFLAGS=-iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64' 'LIBS=-lSystem' 'LDSHARED=xcrun -sdk macosx clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64' 'LDFRAMEWORK=xcrun -sdk macosx clang -dynamiclib -arch arm64 -arch x86_64' 'LDFLAGS=-Wl,-headerpad,0x1000' 'OBJROOT=/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Binaries/python3/install/TempContent/Objects' 'SDKROOT=/AppleInternal/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.Internal.sdk' 'build_alias=x86_64-apple-darwin' 'host_alias=x86_64-apple-darwin'"
	LIBS = "-ldl -lSystem  -framework CoreFoundation"
	SHLIBS = "-ldl -lSystem  -framework CoreFoundation"

rgommers · 2022-08-16T11:48:02Z

Ah, it depends on the Python install:

# conda-forge
% python3 -m sysconfig |grep "SHLIBS"
	SHLIBS = "-ldl   -framework CoreFoundation"

# Apple's Python
% /usr/bin/python3 -m sysconfig |grep "SHLIBS"
	SHLIBS = "-ldl -lSystem  -framework CoreFoundation"

mattip · 2022-08-16T18:18:05Z

Trying to reproduce this on a local M1 machine leads me down a rabbit hole. When I build NumPy with arch -x86_64 python runtests.py, the dragon4.c code does not compile. Somehow npy_float128 is not defined.

mattip · 2022-08-16T19:24:35Z

macOS passed in 07a4352. I am not sure why. Trying again with a little less debugging.

mattip · 2022-08-17T04:38:50Z

CI is passing, twice (both 07a4352 and the latest commit)

rgommers

LGTM, recommend adding a comment but overall good to merge.

rgommers · 2022-08-18T19:03:35Z

azure-pipelines.yml

      displayName: 'Run Full NumPy Test Suite'
      condition: eq(variables['USE_OPENBLAS'], '1')
+      env:
+        LIBRARY_PATH: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib


How about above this line adding:

# The Python install in GitHub Actions adds `-lSystem`, which fails without adding this (see gh-22043)

I don't think it is coming from the macOS-provided python, rather from gcc/gfortran (as opposed to clang, which does not add it). I did not find SHLIBS in NumPy nor in distutils/setuptools, and the stack overflows I linked to did not mention python. Adding a comment in that vein.

makes sense. that's a gfortran that anyway needs an update soon, so hopefully that goes away in the next update

rgommers

All green, so in it goes. Thanks @mattip!

rgommers · 2022-08-24T03:33:11Z

I think this explains what is going on: https://developer.apple.com/forums/thread/655588?answerId=665804022#665804022. The stub library libSystem.tbd indeed exists where I'd expect it based on that explanation.

I don't think it is coming from the macOS-provided python, rather from gcc/gfortran (as opposed to clang, which does not add it).

From the link above: OTOH, if your third-party tools are using their own linker then you’ll have to determine whether that linker has been updated to use stub libraries. So yes, that makes sense. The gfortran we use in this job should be removed soon, it's way too old and now also well below the minimum GCC version we need.

charris · 2022-09-08T20:08:07Z

Marked for backport, looks like this will be needed for 1.23.x soon. It would be good if 12.0 could be made to work, I suspect we will need it soon.

[ci skip]

BUILD: use macos-11 image on azure, macos-1015 is deprecated

1123e29

charris added 03 - Maintenance 36 - Build Build related PR labels Jul 25, 2022

charris changed the title ~~BUILD: use macos-11 image on azure, macos-1015 is deprecated~~ BLD: use macos-11 image on azure, macos-1015 is deprecated Jul 25, 2022

HaoZeke self-assigned this Jul 26, 2022

rgommers mentioned this pull request Aug 11, 2022

MAINT: bump macOS 10.15 to 11 in wheel and testing #22108

Closed

mattip mentioned this pull request Aug 11, 2022

MAINT: Simplify npymath #22090

Merged

add LIBRARY_PATH

81842f6

mattip force-pushed the macos1015 branch from c6cb681 to 81842f6 Compare August 16, 2022 14:57

mattip added 2 commits August 16, 2022 21:21

add debug printing

07a4352

remove part of CI debugging

2e950b2

rgommers approved these changes Aug 18, 2022

View reviewed changes

add comment from review

d7b407a

rgommers approved these changes Aug 19, 2022

View reviewed changes

rgommers merged commit 831c04a into numpy:main Aug 19, 2022

rgommers added this to the 1.24.0 release milestone Aug 19, 2022

charris added the 09 - Backport-Candidate PRs tagged should be backported label Sep 8, 2022

charris modified the milestones: 1.24.0 release, 1.23.4 release Sep 8, 2022

charris pushed a commit to charris/numpy that referenced this pull request Oct 5, 2022

BLD: use macos-11 image on azure, macos-1015 is deprecated (numpy#22043)

6ceca33

[ci skip]

charris mentioned this pull request Oct 5, 2022

BLD: use macos-11 image on azure, macos-1015 is deprecated #22382

Merged

charris removed the 09 - Backport-Candidate PRs tagged should be backported label Oct 5, 2022

charris removed this from the 1.23.4 release milestone Oct 5, 2022

mattip mentioned this pull request Nov 3, 2022

MAINT, BLD: Wheel CI: Update cibuildwheel to 2.11.2 #22524

Merged

mattip deleted the macos1015 branch December 27, 2022 16:50

Uh oh!

Conversation

mattip commented Jul 25, 2022

Uh oh!

rgommers commented Jul 25, 2022

Uh oh!

mattip commented Jul 25, 2022

Uh oh!

rgommers commented Jul 25, 2022

Uh oh!

mattip commented Aug 11, 2022

Uh oh!

rgommers commented Aug 11, 2022

Uh oh!

mattip commented Aug 15, 2022

Uh oh!

rgommers commented Aug 15, 2022

Uh oh!

mattip commented Aug 16, 2022

Uh oh!

rgommers commented Aug 16, 2022

Uh oh!

mattip commented Aug 16, 2022

Uh oh!

mattip commented Aug 16, 2022

Uh oh!

mattip commented Aug 17, 2022

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers Aug 18, 2022

Choose a reason for hiding this comment

Uh oh!

mattip Aug 19, 2022

Choose a reason for hiding this comment

Uh oh!

rgommers Aug 19, 2022

Choose a reason for hiding this comment

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers commented Aug 24, 2022

Uh oh!

charris commented Sep 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants