On a clean Linux system (nothing but the most recent Manjaro Linux and the latest 64-bit Anaconda Linux install) I decided to try to use the conda compilers instead of the system ones to build NumPy and SciPy against the Anaconda-provided Python 3.7. There are some issues with that.
- The
conda activate scripts set LDFLAGS. This needs to be handled by setting export NPY_DISTUTILS_APPEND_FLAGS=1. That's a good thing to put in your .bash_profile anyway. So not a big issue.
- The build fails because of include paths:
In file included from numpy/core/src/common/numpyos.c:23:0:
/home/rgommers/anaconda3/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/xlocale.h:28:16: error: redefinition of 'struct __locale_struct'
typedef struct __locale_struct
^~~~~~~~~~~~~~~
In file included from /usr/include/bits/types/locale_t.h:22:0,
from /usr/include/string.h:152,
from /home/rgommers/anaconda3/include/python3.7m/Python.h:30,
from numpy/core/src/common/numpyos.c:2:
/usr/include/bits/types/__locale_t.h:28:8: note: originally defined here
struct __locale_struct
^~~~~~~~~~~~~~~
In file included from numpy/core/src/common/numpyos.c:23:0:
/home/rgommers/anaconda3/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/xlocale.h:40:4: error: conflicting types for '__locale_t'
} *__locale_t;
^~~~~~~~~~
In file included from /usr/include/bits/types/locale_t.h:22:0,
from /usr/include/string.h:152,
from /home/rgommers/anaconda3/include/python3.7m/Python.h:30,
from numpy/core/src/common/numpyos.c:2:
/usr/include/bits/types/__locale_t.h:42:33: note: previous declaration of '__locale_t' was here
typedef struct __locale_struct *__locale_t;
^~~~~~~~~~
In file included from numpy/core/src/common/numpyos.c:23:0:
/home/rgommers/anaconda3/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/xlocale.h:43:20: error: conflicting types for 'locale_t'
typedef __locale_t locale_t;
^~~~~~~~
In file included from /usr/include/string.h:152:0,
from /home/rgommers/anaconda3/include/python3.7m/Python.h:30,
from numpy/core/src/common/numpyos.c:2:
/usr/include/bits/types/locale_t.h:24:20: note: previous declaration of 'locale_t' was here
typedef __locale_t locale_t;
^~~~~~~~
numpy/core/src/common/numpyos.c: In function 'NumPyOS_ascii_strtold':
numpy/core/src/common/numpyos.c:568:13: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
clocale = newlocale(LC_ALL_MASK, "C", NULL);
^
numpy/core/src/common/numpyos.c:571:39: warning: passing argument 3 of 'strtold_l' from incompatible pointer type [-Wincompatible-pointer-types]
result = strtold_l(s, endptr, clocale);
^~~~~~~
In file included from /home/rgommers/anaconda3/include/python3.7m/Python.h:34:0,
from numpy/core/src/common/numpyos.c:2:
/usr/include/stdlib.h:303:20: note: expected 'locale_t {aka struct __locale_struct *}' but argument is of type 'locale_t {aka struct __locale_struct *}'
extern long double strtold_l (const char *__restrict __nptr,
^~~~~~~~~
numpy/core/src/common/numpyos.c:572:20: warning: passing argument 1 of 'freelocale' from incompatible pointer type [-Wincompatible-pointer-types]
freelocale(clocale);
^~~~~~~
In file included from numpy/core/src/common/numpyos.c:4:0:
/usr/include/locale.h:180:13: note: expected 'locale_t {aka struct __locale_struct *}' but argument is of type 'locale_t {aka struct __locale_struct *}'
extern void freelocale (locale_t __dataset) __THROW;
^~~~~~~~~~
x86_64-conda_cos6-linux-gnu-cc: numpy/core/src/common/npy_longdouble.c
x86_64-conda_cos6-linux-gnu-cc: numpy/core/src/multiarray/mapping.c
x86_64-conda_cos6-linux-gnu-cc: numpy/core/src/multiarray/methods.c
x86_64-conda_cos6-linux-gnu-cc: numpy/core/src/umath/matmul.c
x86_64-conda_cos6-linux-gnu-cc: numpy/core/src/umath/ufunc_object.c
error: Command "/home/rgommers/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DSCIPY_MKL_H -DHAVE_CBLAS -I/usr/local/include -I/usr/include -I/home/rgommers/anaconda3/include -Inumpy/core/src/umath -Inumpy/core/src/npymath -Inumpy/core/src/common -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/rgommers/anaconda3/include/python3.7m -Inumpy/core/src/common -Inumpy/core/src/npymath -Inumpy/core/src/common -Inumpy/core/src/npymath -c numpy/core/src/common/numpyos.c -o build/temp.linux-x86_64-3.7/numpy/core/src/common/numpyos.o -MMD -MF build/temp.linux-x86_64-3.7/numpy/core/src/common/numpyos.o.d" failed with exit status 1
The problem seems to be that we explicitly include /usr/include/ and /usr/local/include/ in the default_include_dirs in numpy/distutils/system_info.py. If we remove those, then everything builds fine. I think this is the right thing to do, because those paths are already
included by compilers anyway.
The conda compiler docs linked above don't discuss this. They do explain this:
Some users want to use the latest Anaconda packages but do not yet want to use the Anaconda compilers. To enable this, the latest Python package builds have a default _sysconfigdata file. This file sets the compilers provided by the system, such as gcc and g++, as the default compilers. This way legacy recipes will keep working.
It continues to talk about how to switch to conda compilers by setting _PYTHON_SYSCONFIGDATA_NAME, however this isn't enough.
This is the default (from anaconda3/lib/python3.7/distutils/sysconfig.py):
export _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_x86_64-linux-gnu
So setting it to that is a do-nothing. If you get the name wrong you'll get an error like this:
ModuleNotFoundError: No module named '_sysconfigdata_m_linux_x86_64-linux-gn'
Now we set it to the value as set by anaconda3/etc/conda/activate.d/activate-gcc_linux-64.sh:
export _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
This is what the conda compiler docs recommend:
https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#backward-compatibility.
It still gives us the same error though:
error: Command "/home/rgommers/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DSCIPY_MKL_H -DHAVE_CBLAS -I/usr/local/include -I/usr/include -I/home/rgommers/anaconda3/include -Inumpy/core/src/umath -Inumpy/core/src/npymath -Inumpy/core/src/common -Inumpy/core/include -Inumpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/rgommers/anaconda3/include/python3.7m -Inumpy/core/src/common -Inumpy/core/src/npymath -Inumpy/core/src/common -Inumpy/core/src/npymath -c numpy/core/src/common/numpyos.c -o build/temp.linux-x86_64-3.7/numpy/core/src/common/numpyos.o -MMD -MF build/temp.linux-x86_64-3.7/numpy/core/src/common/numpyos.o.d" failed with exit status 1
Also axplicitly including --sysroot=/home/rgommers/anaconda3/x86_64-conda_cos6-linux-gnu/sysroot in that failing build command doesn't change the build error.
On the other hand, this fixes things (both with and without setting _PYTHON_SYSCONFIGDATA_NAME):
diff --git a/numpy/distutils/system_info.py b/numpy/distutils/system_info.py
index 4d923ad26..8d32db6f2 100644
--- a/numpy/distutils/system_info.py
+++ b/numpy/distutils/system_info.py
@@ -278,7 +278,7 @@ else:
'/opt/local/lib', '/sw/lib'], platform_bits)
default_runtime_dirs = []
default_include_dirs = ['/usr/local/include',
- '/opt/include', '/usr/include',
+ '/opt/include',
# path of umfpack under macports
'/opt/local/include/ufsparse',
'/opt/local/include', '/sw/include',
I think that's the right fix (ref https://gcc.gnu.org/onlinedocs/cpp/Search-Path.html). I think that all compilers include /usr/include/ in their search path (however not 100% sure).
On a clean Linux system (nothing but the most recent Manjaro Linux and the latest 64-bit Anaconda Linux install) I decided to try to use the conda compilers instead of the system ones to build NumPy and SciPy against the Anaconda-provided Python 3.7. There are some issues with that.
conda activatescripts setLDFLAGS. This needs to be handled by settingexport NPY_DISTUTILS_APPEND_FLAGS=1. That's a good thing to put in your.bash_profileanyway. So not a big issue.The problem seems to be that we explicitly include
/usr/include/and/usr/local/include/in thedefault_include_dirsinnumpy/distutils/system_info.py. If we remove those, then everything builds fine. I think this is the right thing to do, because those paths are alreadyincluded by compilers anyway.
The conda compiler docs linked above don't discuss this. They do explain this:
Some users want to use the latest Anaconda packages but do not yet want to use the Anaconda compilers. To enable this, the latest Python package builds have a default _sysconfigdata file. This file sets the compilers provided by the system, such as gcc and g++, as the default compilers. This way legacy recipes will keep working.
It continues to talk about how to switch to conda compilers by setting
_PYTHON_SYSCONFIGDATA_NAME, however this isn't enough.This is the default (from
anaconda3/lib/python3.7/distutils/sysconfig.py):So setting it to that is a do-nothing. If you get the name wrong you'll get an error like this:
Now we set it to the value as set by
anaconda3/etc/conda/activate.d/activate-gcc_linux-64.sh:This is what the conda compiler docs recommend:
https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#backward-compatibility.
It still gives us the same error though:
Also axplicitly including
--sysroot=/home/rgommers/anaconda3/x86_64-conda_cos6-linux-gnu/sysrootin that failing build command doesn't change the build error.On the other hand, this fixes things (both with and without setting
_PYTHON_SYSCONFIGDATA_NAME):I think that's the right fix (ref https://gcc.gnu.org/onlinedocs/cpp/Search-Path.html). I think that all compilers include
/usr/include/in their search path (however not 100% sure).