BUILD: Remove Accelerate for macOS by mattip · Pull Request #14880 · numpy/numpy

mattip · 2019-11-11T21:32:17Z

Accelerate is buggy, closed-source, and not supported. Remove it.

rgommers

+1 for dropping Accelerate support in NumPy. Please propose it on the mailing list though, since it may affect people (at least they need to change their build setups to pull in OpenBLAS somehow).

Also, please undo all the numpy.distutils changes. There are very likely other projects that rely on Accelerate, you can't just remove that.

eric-wieser · 2019-11-11T23:52:38Z

I thought we dropped support for accelerate ages ago - at least, I remember the mailing list discussion, and my changes to _linalg_umath relied on a lapack with bugfixes not present in accelerate.

numpy/distutils/system_info.py

mattip · 2019-11-12T00:34:19Z

reverted changes to distutils

numpy/_build_utils/apple_accelerate.py

doc/source/user/building.rst

eric-wieser · 2019-11-12T09:29:54Z

numpy/core/setup.py

+    # Accelerate is buggy and not supported
+    if 'accelerate' in blas_opt_info.blas_order:
+        blas_opt_info.blas_order.remove('accelerate')


I'm tempted to remove it from lapack here too, to keep it in one place

makes sense

You mean move the corresponding change to lapack_opt_info out of numpy/linalg.setup.py to here? That seems fragile.

I repeated the code in both numpy/core/setup.py and numpy/linalg/setup.py, with comments.

rgommers

This looks good to me now

WarrenWeckesser · 2019-11-16T07:59:31Z

The note about Accelerate in INSTALL.rst.txt at https://github.com/numpy/numpy/blob/master/INSTALL.rst.txt#os-x should be updated or removed.

WarrenWeckesser · 2019-11-16T08:19:01Z

I'm using Mac OSX 10.12.6. I built Python 3.8 from source, and then tried to install NumPy from this pull request. I did not create the file 'site.cfg'. ~~No optimized BLAS libraries are installed.~~ Other than the libraries provided by the Accelerate framework, no optimized BLAS librarues are installed. The build failed (see below). Is this supposed to work? If not, the "Prerequisites" section of INSTALL.rst.txt should be updated to explicitly state the requirement for a preinstalled BLAS library, and the error message in the UserWarning seen below should say something about the BLAS requirement, and not "Optimized (vendor) Blas libraries are not found. Falls back to netlib Blas library which has worse performance."

$ python3 setup.py install

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last NumPy release on PyPi)


Running from numpy source directory.
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_bit_generator.pyx
Processing numpy/random/_bounded_integers.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_common.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_philox.pyx
Processing numpy/random/_sfc64.pyx
Processing numpy/random/mtrand.pyx
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries tatlas not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
  libraries satlas not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in ['/Users/warren/local-py38/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

/Users/warren/repos/git/forks/numpy/numpy/distutils/system_info.py:697: UserWarning: 
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  self.calc_info()
blas_info:
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99

creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var
creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var/folders
creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var/folders/p9
creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn
creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T
creating /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r
compile options: '-I/usr/local/include -I/usr/include -I/Users/warren/local-py38/include -c'
gcc: /var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/source.c
/var/folders/p9/j_v_3wvd2bx5ftq544pvvf1w0000gn/T/tmp5pv1os2r/source.c:1:10: fatal error: 'cblas.h' file not found
#include <cblas.h>
         ^~~~~~~~~
1 error generated.
  FOUND:
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/local/include', '/usr/include', '/Users/warren/local-py38/include']

  FOUND:
    define_macros = [('NO_ATLAS_INFO', 1)]
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/local/include', '/usr/include', '/Users/warren/local-py38/include']

non-existing path in 'numpy/distutils': 'site.cfg'
Traceback (most recent call last):
  File "setup.py", line 450, in <module>
    setup_package()
  File "setup.py", line 442, in setup_package
    setup(**metadata)
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/core.py", line 137, in setup
    config = configuration()
  File "setup.py", line 164, in configuration
    config.add_subpackage('numpy')
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 1033, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 999, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/setup.py", line 16, in configuration
    config.add_subpackage('linalg')
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 1033, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 999, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/Users/warren/repos/git/forks/numpy/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/linalg/setup.py", line 14, in configuration
    for opt_order in (blas_opt_info.blas_order, lapack_opt_info.lapack_order):
NameError: name 'blas_opt_info' is not defined

WarrenWeckesser · 2019-11-16T08:33:40Z

Update: I said

No optimized BLAS libraries are installed.

but that's not true. /usr/lib contains libblas.dylib and libcblas.dylib, which are symbolic links to the implementations in the Accelerate Framework. So perhaps the build system is "confused" about finding BLAS in /usr/lib, but not finding the corresponding header files in /usr/include.

mattip · 2019-11-18T12:07:22Z

I think your log is OK; since the blas_info is not setting define_macros = [('HAVE_CBLAS', None)] it should not link to any blas backend. I have fixed the linalg failure, could you try again and verify that _multiarray_umath.dylib does not link to the accelerate or to the symbolic links?

WarrenWeckesser · 2019-11-18T19:12:08Z

@mattip, thanks, I can build this PR now. I used otool -L to check the following in the installed numpy package:

In numpy/core:

$ otool -L _multiarray_umath.cpython-38-darwin.so 
_multiarray_umath.cpython-38-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

In numpy/linalg:

$ otool -L _umath_linalg.cpython-38-darwin.so 
_umath_linalg.cpython-38-darwin.so:
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

So it looks like linalg is still linking to Accelerate. This might be a quirk of how my system is set up. Let me know if there is anything I should check before digging into this any deeper.

WarrenWeckesser · 2019-11-18T19:24:04Z

The links to Accelerate in _umath_linalg.cpython-38-darwin.so are the destinations of the symbolic links /usr/lib/libblas.dylib and /usr/lib/liblapack.dylib that I mentioned in a previous comment.

mattip · 2019-11-18T19:35:15Z

Does it also build like that on scipy? I think they removed support for accelerate as well.
Can you put your build.log file on a public pastebin or gist somewhere?

WarrenWeckesser · 2019-11-18T20:01:20Z

Does it also build like that on scipy? I think they removed support for accelerate as well.

I haven't yet tried building scipy with no optimized blas installed -- I usually work from a miniconda installation, which includes MKL. I'll try a clean(ish) build, similar to what I'm doing with numpy.

Can you put your build.log file on a public pastebin or gist somewhere?

Here you go: https://gist.github.com/WarrenWeckesser/200b6d91e34a41dd2fe90258532a76ac

WarrenWeckesser · 2019-11-18T20:05:25Z

And here's np.show_config() from the same build:

~$ python3
Python 3.8.0 (default, Nov 16 2019, 02:34:22) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.18.0.dev0+5fa589c'
>>> np.show_config()
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
blas_info:
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/local/include', '/usr/include', '/Users/warren/local-py38/include']
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1)]
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib']
    include_dirs = ['/usr/local/include', '/usr/include', '/Users/warren/local-py38/include']
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
flame_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_info:
    libraries = ['lapack', 'lapack']
    library_dirs = ['/usr/lib']
    language = f77
lapack_opt_info:
    libraries = ['lapack', 'lapack', 'blas', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
    include_dirs = ['/usr/local/include', '/usr/include', '/Users/warren/local-py38/include']

WarrenWeckesser · 2019-11-19T06:06:49Z

Based on this:

In [2]: from numpy.distutils.system_info import get_info                                                                                 

In [3]: get_info('lapack_opt', 0)                                                                                                        
Out[3]: 
{'extra_compile_args': ['-msse3',
  '-I/System/Library/Frameworks/vecLib.framework/Headers'],
 'extra_link_args': ['-Wl,-framework', '-Wl,Accelerate'],
 'define_macros': [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]}

I tried an experiment in which I made this change to numpy/linalg/setup.py:

$ git diff
diff --git a/numpy/linalg/setup.py b/numpy/linalg/setup.py
index e8ba315d5..75bec30d6 100644
--- a/numpy/linalg/setup.py
+++ b/numpy/linalg/setup.py
@@ -32,7 +32,8 @@ def configuration(parent_package='', top_path=None):
     ]
     all_sources = config.paths(lapack_lite_src)
 
-    lapack_info = get_info('lapack_opt', 0)  # and {}
+    #lapack_info = get_info('lapack_opt', 0)  # and {}
+    lapack_info = {}
 
     def get_lapack_lite_sources(ext, build_dir):
         if not lapack_info:

In other words, just set lapack_info to {} instead of calling get_info('lapack_opt', 0).
When I build NumPy from this PR with that change included, the compiled .so files in numpy/linalg in the installation directory are no longer linked to Accelerate:


$ otool -L *.so
_umath_linalg.cpython-38-darwin.so:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
lapack_lite.cpython-38-darwin.so:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

and the full test suite still passes. So it is possible to make this work, but I haven't figured out enough of the numpy.distutils code to know what the correct change should be.

WarrenWeckesser · 2019-11-21T19:04:20Z

When I run the test suite for this PR with Python 3.8 built from source (still using Mac OSX 10.12.6), I get one test failure:

_____________________________________________________________________________ TestPolynomial.test_polyfit ______________________________________________________________________________

self = <numpy.lib.tests.test_polynomial.TestPolynomial object at 0x134ab3190>

    def test_polyfit(self):
        c = np.array([3., 2., 1.])
        x = np.linspace(0, 2, 7)
        y = np.polyval(c, x)
        err = [1, -1, 1, -1, 1, -1, 1]
        weights = np.arange(8, 1, -1)**2/7.0
    
        # Check exception when too few points for variance estimate. Note that
        # the estimate requires the number of data points to exceed
        # degree + 1
        assert_raises(ValueError, np.polyfit,
                      [1], [1], deg=0, cov=True)
    
        # check 1D case
        m, cov = np.polyfit(x, y+err, 2, cov=True)
        est = [3.8571, 0.2857, 1.619]
        assert_almost_equal(est, m, decimal=4)
        val0 = [[ 1.4694, -2.9388,  0.8163],
                [-2.9388,  6.3673, -2.1224],
                [ 0.8163, -2.1224,  1.161 ]]
        assert_almost_equal(val0, cov, decimal=4)
    
        m2, cov2 = np.polyfit(x, y+err, 2, w=weights, cov=True)
        assert_almost_equal([4.8927, -1.0177, 1.7768], m2, decimal=4)
        val = [[ 4.3964, -5.0052,  0.4878],
               [-5.0052,  6.8067, -0.9089],
               [ 0.4878, -0.9089,  0.3337]]
        assert_almost_equal(val, cov2, decimal=4)
    
        m3, cov3 = np.polyfit(x, y+err, 2, w=weights, cov="unscaled")
        assert_almost_equal([4.8927, -1.0177, 1.7768], m3, decimal=4)
        val = [[ 0.1473, -0.1677,  0.0163],
               [-0.1677,  0.228 , -0.0304],
               [ 0.0163, -0.0304,  0.0112]]
        assert_almost_equal(val, cov3, decimal=4)
    
        # check 2D (n,1) case
        y = y[:, np.newaxis]
        c = c[:, np.newaxis]
        assert_almost_equal(c, np.polyfit(x, y, 2))
        # check 2D (n,2) case
        yy = np.concatenate((y, y), axis=1)
        cc = np.concatenate((c, c), axis=1)
        assert_almost_equal(cc, np.polyfit(x, yy, 2))
    
        m, cov = np.polyfit(x, yy + np.array(err)[:, np.newaxis], 2, cov=True)
        assert_almost_equal(est, m[:, 0], decimal=4)
        assert_almost_equal(est, m[:, 1], decimal=4)
        assert_almost_equal(val0, cov[:, :, 0], decimal=4)
        assert_almost_equal(val0, cov[:, :, 1], decimal=4)
    
        # check order 1 (deg=0) case, were the analytic results are simple
        np.random.seed(123)
        y = np.random.normal(size=(4, 10000))
>       mean, cov = np.polyfit(np.zeros(y.shape[0]), y, deg=0, cov=True)

c          = array([[3.],
       [2.],
       [1.]])
cc         = array([[3., 3.],
       [2., 2.],
       [1., 1.]])
cov        = array([[[ 1.46938776,  1.46938776],
        [-2.93877551, -2.93877551],
        [ 0.81632653,  0.81632653]],

       [...44898]],

       [[ 0.81632653,  0.81632653],
        [-2.12244898, -2.12244898],
        [ 1.16099773,  1.16099773]]])
cov2       = array([[ 4.39644554, -5.00516045,  0.48778036],
       [-5.00516045,  6.80672276, -0.90891376],
       [ 0.48778036, -0.90891376,  0.33367938]])
cov3       = array([[ 0.14727151, -0.16766216,  0.0163396 ],
       [-0.16766216,  0.22801064, -0.03044666],
       [ 0.0163396 , -0.03044666,  0.01117754]])
err        = [1, -1, 1, -1, 1, -1, ...]
est        = [3.8571, 0.2857, 1.619]
m          = array([[3.85714286, 3.85714286],
       [0.28571429, 0.28571429],
       [1.61904762, 1.61904762]])
m2         = array([ 4.89274172, -1.01769136,  1.77675979])
m3         = array([ 4.89274172, -1.01769136,  1.77675979])
self       = <numpy.lib.tests.test_polynomial.TestPolynomial object at 0x134ab3190>
val        = [[0.1473, -0.1677, 0.0163], [-0.1677, 0.228, -0.0304], [0.0163, -0.0304, 0.0112]]
val0       = [[1.4694, -2.9388, 0.8163], [-2.9388, 6.3673, -2.1224], [0.8163, -2.1224, 1.161]]
weights    = array([9.14285714, 7.        , 5.14285714, 3.57142857, 2.28571429,
       1.28571429, 0.57142857])
x          = array([0.        , 0.33333333, 0.66666667, 1.        , 1.33333333,
       1.66666667, 2.        ])
y          = array([[-1.0856306 ,  0.99734545,  0.2829785 , ..., -0.47226641,
         0.58196437,  0.97061286],
       [-1.2409696...164,  1.65991687],
       [-1.05417044, -0.78301134,  1.82790084, ...,  0.52562332,
         0.99480016, -0.80351273]])
yy         = array([[ 1.        ,  1.        ],
       [ 2.        ,  2.        ],
       [ 3.66666667,  3.66666667],
       [ 6.  ... 6.        ],
       [ 9.        ,  9.        ],
       [12.66666667, 12.66666667],
       [17.        , 17.        ]])

/Users/warren/local-py38/lib/python3.8/site-packages/numpy-1.18.0.dev0+5fa589c-py3.8-macosx-10.12-x86_64.egg/numpy/lib/tests/test_polynomial.py:171: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
<__array_function__ internals>:5: in polyfit
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([0., 0., 0., 0.])
y = array([[-1.0856306 ,  0.99734545,  0.2829785 , ..., -0.47226641,
         0.58196437,  0.97061286],
       [-1.2409696...164,  1.65991687],
       [-1.05417044, -0.78301134,  1.82790084, ...,  0.52562332,
         0.99480016, -0.80351273]])
deg = 0, rcond = 8.881784197001252e-16, full = False, w = None, cov = True

    @array_function_dispatch(_polyfit_dispatcher)
    def polyfit(x, y, deg, rcond=None, full=False, w=None, cov=False):
        """
        Least squares polynomial fit.
    
        Fit a polynomial ``p(x) = p[0] * x**deg + ... + p[deg]`` of degree `deg`
        to points `(x, y)`. Returns a vector of coefficients `p` that minimises
        the squared error in the order `deg`, `deg-1`, ... `0`.
    
        The `Polynomial.fit <numpy.polynomial.polynomial.Polynomial.fit>` class
        method is recommended for new code as it is more stable numerically. See
        the documentation of the method for more information.
    
        Parameters
        ----------
        x : array_like, shape (M,)
            x-coordinates of the M sample points ``(x[i], y[i])``.
        y : array_like, shape (M,) or (M, K)
            y-coordinates of the sample points. Several data sets of sample
            points sharing the same x-coordinates can be fitted at once by
            passing in a 2D-array that contains one dataset per column.
        deg : int
            Degree of the fitting polynomial
        rcond : float, optional
            Relative condition number of the fit. Singular values smaller than
            this relative to the largest singular value will be ignored. The
            default value is len(x)*eps, where eps is the relative precision of
            the float type, about 2e-16 in most cases.
        full : bool, optional
            Switch determining nature of return value. When it is False (the
            default) just the coefficients are returned, when True diagnostic
            information from the singular value decomposition is also returned.
        w : array_like, shape (M,), optional
            Weights to apply to the y-coordinates of the sample points. For
            gaussian uncertainties, use 1/sigma (not 1/sigma**2).
        cov : bool or str, optional
            If given and not `False`, return not just the estimate but also its
            covariance matrix. By default, the covariance are scaled by
            chi2/sqrt(N-dof), i.e., the weights are presumed to be unreliable
            except in a relative sense and everything is scaled such that the
            reduced chi2 is unity. This scaling is omitted if ``cov='unscaled'``,
            as is relevant for the case that the weights are 1/sigma**2, with
            sigma known to be a reliable estimate of the uncertainty.
    
        Returns
        -------
        p : ndarray, shape (deg + 1,) or (deg + 1, K)
            Polynomial coefficients, highest power first.  If `y` was 2-D, the
            coefficients for `k`-th data set are in ``p[:,k]``.
    
        residuals, rank, singular_values, rcond
            Present only if `full` = True.  Residuals is sum of squared residuals
            of the least-squares fit, the effective rank of the scaled Vandermonde
            coefficient matrix, its singular values, and the specified value of
            `rcond`. For more details, see `linalg.lstsq`.
    
        V : ndarray, shape (M,M) or (M,M,K)
            Present only if `full` = False and `cov`=True.  The covariance
            matrix of the polynomial coefficient estimates.  The diagonal of
            this matrix are the variance estimates for each coefficient.  If y
            is a 2-D array, then the covariance matrix for the `k`-th data set
            are in ``V[:,:,k]``
    
    
        Warns
        -----
        RankWarning
            The rank of the coefficient matrix in the least-squares fit is
            deficient. The warning is only raised if `full` = False.
    
            The warnings can be turned off by
    
            >>> import warnings
            >>> warnings.simplefilter('ignore', np.RankWarning)
    
        See Also
        --------
        polyval : Compute polynomial values.
        linalg.lstsq : Computes a least-squares fit.
        scipy.interpolate.UnivariateSpline : Computes spline fits.
    
        Notes
        -----
        The solution minimizes the squared error
    
        .. math ::
            E = \\sum_{j=0}^k |p(x_j) - y_j|^2
    
        in the equations::
    
            x[0]**n * p[0] + ... + x[0] * p[n-1] + p[n] = y[0]
            x[1]**n * p[0] + ... + x[1] * p[n-1] + p[n] = y[1]
            ...
            x[k]**n * p[0] + ... + x[k] * p[n-1] + p[n] = y[k]
    
        The coefficient matrix of the coefficients `p` is a Vandermonde matrix.
    
        `polyfit` issues a `RankWarning` when the least-squares fit is badly
        conditioned. This implies that the best fit is not well-defined due
        to numerical error. The results may be improved by lowering the polynomial
        degree or by replacing `x` by `x` - `x`.mean(). The `rcond` parameter
        can also be set to a value smaller than its default, but the resulting
        fit may be spurious: including contributions from the small singular
        values can add numerical noise to the result.
    
        Note that fitting polynomial coefficients is inherently badly conditioned
        when the degree of the polynomial is large or the interval of sample points
        is badly centered. The quality of the fit should always be checked in these
        cases. When polynomial fits are not satisfactory, splines may be a good
        alternative.
    
        References
        ----------
        .. [1] Wikipedia, "Curve fitting",
               https://en.wikipedia.org/wiki/Curve_fitting
        .. [2] Wikipedia, "Polynomial interpolation",
               https://en.wikipedia.org/wiki/Polynomial_interpolation
    
        Examples
        --------
        >>> import warnings
        >>> x = np.array([0.0, 1.0, 2.0, 3.0,  4.0,  5.0])
        >>> y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
        >>> z = np.polyfit(x, y, 3)
        >>> z
        array([ 0.08703704, -0.81349206,  1.69312169, -0.03968254]) # may vary
    
        It is convenient to use `poly1d` objects for dealing with polynomials:
    
        >>> p = np.poly1d(z)
        >>> p(0.5)
        0.6143849206349179 # may vary
        >>> p(3.5)
        -0.34732142857143039 # may vary
        >>> p(10)
        22.579365079365115 # may vary
    
        High-order polynomials may oscillate wildly:
    
        >>> with warnings.catch_warnings():
        ...     warnings.simplefilter('ignore', np.RankWarning)
        ...     p30 = np.poly1d(np.polyfit(x, y, 30))
        ...
        >>> p30(4)
        -0.80000000000000204 # may vary
        >>> p30(5)
        -0.99999999999999445 # may vary
        >>> p30(4.5)
        -0.10547061179440398 # may vary
    
        Illustration:
    
        >>> import matplotlib.pyplot as plt
        >>> xp = np.linspace(-2, 6, 100)
        >>> _ = plt.plot(x, y, '.', xp, p(xp), '-', xp, p30(xp), '--')
        >>> plt.ylim(-2,2)
        (-2, 2)
        >>> plt.show()
    
        """
        order = int(deg) + 1
        x = NX.asarray(x) + 0.0
        y = NX.asarray(y) + 0.0
    
        # check arguments.
        if deg < 0:
            raise ValueError("expected deg >= 0")
        if x.ndim != 1:
            raise TypeError("expected 1D vector for x")
        if x.size == 0:
            raise TypeError("expected non-empty vector for x")
        if y.ndim < 1 or y.ndim > 2:
            raise TypeError("expected 1D or 2D array for y")
        if x.shape[0] != y.shape[0]:
            raise TypeError("expected x and y to have same length")
    
        # set rcond
        if rcond is None:
            rcond = len(x)*finfo(x.dtype).eps
    
        # set up least squares equation for powers of x
        lhs = vander(x, order)
        rhs = y
    
        # apply weighting
        if w is not None:
            w = NX.asarray(w) + 0.0
            if w.ndim != 1:
                raise TypeError("expected a 1-d array for weights")
            if w.shape[0] != y.shape[0]:
                raise TypeError("expected w and y to have the same length")
            lhs *= w[:, NX.newaxis]
            if rhs.ndim == 2:
                rhs *= w[:, NX.newaxis]
            else:
                rhs *= w
    
        # scale lhs to improve condition number and solve
        scale = NX.sqrt((lhs*lhs).sum(axis=0))
        lhs /= scale
        c, resids, rank, s = lstsq(lhs, rhs, rcond)
        c = (c.T/scale).T  # broadcast scale coefficients
    
        # warn on rank reduction, which indicates an ill conditioned matrix
        if rank != order and not full:
            msg = "Polyfit may be poorly conditioned"
>           warnings.warn(msg, RankWarning, stacklevel=4)
E           numpy.RankWarning: Polyfit may be poorly conditioned

c          = array([[0.00000000e+000, 3.45845952e-323, 0.00000000e+000, ...,
        2.26526868e-001, 5.48309104e-002, 2.05365551e-001]])
cov        = True
deg        = 0
full       = False
lhs        = array([[0.5],
       [0.5],
       [0.5],
       [0.5]])
msg        = 'Polyfit may be poorly conditioned'
order      = 1
rank       = 3
rcond      = 8.881784197001252e-16
resids     = array([], dtype=float64)
rhs        = array([[-1.0856306 ,  0.99734545,  0.2829785 , ..., -0.47226641,
         0.58196437,  0.97061286],
       [-1.2409696...164,  1.65991687],
       [-1.05417044, -0.78301134,  1.82790084, ...,  0.52562332,
         0.99480016, -0.80351273]])
s          = array([1.])
scale      = array([2.])
w          = None
x          = array([0., 0., 0., 0.])
y          = array([[-1.0856306 ,  0.99734545,  0.2829785 , ..., -0.47226641,
         0.58196437,  0.97061286],
       [-1.2409696...164,  1.65991687],
       [-1.05417044, -0.78301134,  1.82790084, ...,  0.52562332,
         0.99480016, -0.80351273]])

/Users/warren/local-py38/lib/python3.8/site-packages/numpy-1.18.0.dev0+5fa589c-py3.8-macosx-10.12-x86_64.egg/numpy/lib/polynomial.py:637: RankWarning
--------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------
python3(76547,0x10642e3c0) malloc: *** mach_vm_map(size=18446744071926976512) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
init_dgelsd failed init
1 failed, 10302 passed, 81 skipped, 16 xfailed, 3 xpassed in 322.14s (0:05:22)

WarrenWeckesser · 2019-11-21T19:04:56Z

SciPy also removed the use of Accelerate, so as a sanity check, I built SciPy with Python 3.8 built from source, and NumPy 1.17.4 installed with pip (which uses a self-contained copy of openblas). Running otool -L on the BLAS and LAPACK .so files built for scipy.linalg also showed links to the Accelerate framework. So either my system configuration is wonky (technical term, there), or this PR is working as well as can be expected at the moment.

WarrenWeckesser · 2019-11-21T19:07:37Z

It looks like there are still some unknown issues here. After consultation with @mattip, I'm closing the PR, but anyone who has insights into this issue is welcome to comment.

rgommers · 2019-12-23T05:51:24Z

I'm updating the install docs, and was looking at the state of this. @WarrenWeckesser or @mattip can you please expand on why this was closed? We still want this right? A better description and ensuring that this PR is linked from the tracking issue (creating it if it doesn't exist yet) would be very useful to ensure we don't reinvent this particular wheel in the future.

WarrenWeckesser · 2019-12-23T10:40:43Z

The goal of the PR was to disable the use of Accelerate. The problem is that I when I built NumPy from source with the changes in this PR (on macOS 10.12.6), there were still links to Accelerate from the NumPy extension modules. This appears to be the result of /usr/lib/libblas.dylib and /usr/lib/liblapack.dylib being symlinks to the Accelerate framework.

pv · 2019-12-23T10:47:54Z

I remember someone tested that what Scipy does prevents Accelerate from being linked in, and found that it works. Are the libblas.dylib / liblapack.dylib standard parts of an OSX out of box installation?

I imagine what is happening in your case is that they are getting picked up as a generic BLAS and LAPACK libraries.

BUILD: Remove Accelerate for macOS

8bd9dbf

mattip added the component: build label Nov 11, 2019

DOC: add release note

c744daa

rgommers requested changes Nov 11, 2019

View reviewed changes

eric-wieser reviewed Nov 11, 2019

View reviewed changes

numpy/distutils/system_info.py Show resolved Hide resolved

MAINT: revert removal of Accelerate from system_info

7218291

eric-wieser reviewed Nov 12, 2019

View reviewed changes

numpy/_build_utils/apple_accelerate.py Show resolved Hide resolved

eric-wieser reviewed Nov 12, 2019

View reviewed changes

doc/source/user/building.rst Show resolved Hide resolved

mattip added 3 commits November 11, 2019 17:51

MAINT: remove no-longer needed _build_utils

e5e914b

BUILD: remove discovery of accelerate from blas and lapack

6cb887f

BUILD: remove accelerte from CI

f9d0957

eric-wieser requested a review from rgommers November 12, 2019 09:27

eric-wieser reviewed Nov 12, 2019

View reviewed changes

rgommers approved these changes Nov 12, 2019

View reviewed changes

mattip force-pushed the remove-accelerate branch from 5617218 to c05bde1 Compare November 14, 2019 22:40

MAINT: fix from review: remove accelerate from both opt_order occurances

5fa589c

mattip force-pushed the remove-accelerate branch from c05bde1 to 5fa589c Compare November 18, 2019 11:56

WarrenWeckesser closed this Nov 21, 2019

mattip mentioned this pull request Dec 8, 2019

ma.TestPolynomial fails using Accelerate on macOS #14644

Closed

vrakesh mentioned this pull request Feb 26, 2020

ENH: Check for buggy Accelerate on MacOS #15647

Closed

mattip mentioned this pull request Mar 15, 2020

BUILD: Remove Accelerate support #15759

Merged

Uh oh!

Conversation

mattip commented Nov 11, 2019

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Nov 11, 2019

Uh oh!

Uh oh!

mattip commented Nov 12, 2019

Uh oh!

Uh oh!

Uh oh!

eric-wieser Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

rgommers Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

mattip Nov 13, 2019

Choose a reason for hiding this comment

Uh oh!

mattip Nov 13, 2019

Choose a reason for hiding this comment

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

WarrenWeckesser commented Nov 16, 2019

Uh oh!

WarrenWeckesser commented Nov 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WarrenWeckesser commented Nov 16, 2019

Uh oh!

mattip commented Nov 18, 2019

Uh oh!

WarrenWeckesser commented Nov 18, 2019

Uh oh!

WarrenWeckesser commented Nov 18, 2019

Uh oh!

mattip commented Nov 18, 2019

Uh oh!

WarrenWeckesser commented Nov 18, 2019

Uh oh!

WarrenWeckesser commented Nov 18, 2019

Uh oh!

WarrenWeckesser commented Nov 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WarrenWeckesser commented Nov 21, 2019

Uh oh!

WarrenWeckesser commented Nov 21, 2019

Uh oh!

WarrenWeckesser commented Nov 21, 2019

Uh oh!

rgommers commented Dec 23, 2019

Uh oh!

WarrenWeckesser commented Dec 23, 2019

Uh oh!

pv commented Dec 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

WarrenWeckesser commented Nov 16, 2019 •

edited

Loading

WarrenWeckesser commented Nov 19, 2019 •

edited

Loading