Conversation
The LAPACK bundled with Accelerate on OSX 10.7+ makes ARPACK produce invalid results in some cases. This commit disables linking with the Accelerate framework or the LAPACK component, but allows linking with the BLAS one.
Currently, Scipy contains some ABI compatibility wrappers for a few of the BLAS routines, and a few LAPACK routines. Nevertheless, libraries in Scipy (e.g. ARPACK) use parts of LAPACK and BLAS not covered. Moreover, especially on OSX it can be easy for the user to mistakenly link Accelerate (g77 ABI) with an ABI-incompatible LAPACK library. For these reasons, it is better to be strict with the ABI compatibility for the moment.
|
(Travis failure is unrelated; the build sometimes takes > 10 minutes and hits a cutoff. The runtests.py script should maybe produce some output when building...) |
|
Is this pull request still relevant, given that the ARPACK problem has now clearly been identified as a bug in ARPACK? I find removing Accelerate support from scipy a bit extreme - Accelerate is not bad (not great either). I have a certain interest to have a robust scipy installation for Mac users (it's an essential ingredient of a scientific python package we want to release soon), and would be willing to spend some time to work on the ABI wrappers (I think mainly the functions returning REAL are missing - DOUBLE shouldn't cause problems). In any case, why is "-ff2c" not sufficient (in that case one could just ditch the ABI wrappers)? |
|
This PR is not as relevant any more, now that we know the bug is in ARPACK, and know how to fix it (I'll file a PR for the ARPACK patch, a more complete fix will hopefully be included also in future arpack-ng versions). I would still prefer to include the parts in this PR that check for BLAS+LAPACK ABI compatibility, as long as the ABI wrapper coverage is not 100%. What I know is currently missing, and used e.g. in ARPACK, is: The ABI wrappers were AFAIK introduced because they make installation more straightforward, without needing to mess as much with compiler flags. In my opinion, however, it's the user's responsibility to make sure their Fortran compiler environment is sane. I would prefer dropping them. If you use |
|
Closing, I'll submit a reworked PR for the ABI checks only. |
The LAPACK bundled with Accelerate on OSX 10.7+ causes problems for ARPACK. This is either a bug in ARPACK or Accelerate, but in the meantime we need to do something in Scipy, as I doubt the problem will be resolved any time soon.
What is done here:
The point (1) is necessary because of the unresolved incompatibility between ARPACK and Accelerate.
The point (2) is required because the coverage of Fortran ABI wrappers currently in Scipy master is insufficient --- many LAPACK and BLAS routines that are susceptible to Fortran ABI incompatibilities are not called via a C-API.
Another point for (2) is that on OSX it is easy for an unsuspecting user to use a LAPACK library compiled with wrong compiler flags.
These ABI restrictions affect only building Scipy and don't infect 3rd party packages --- which Fortran ABI a certain Python module is compiled against does not affect other Python modules.
The practical effect on OSX users who wish to use the BLAS part of Accelerate is that they need to download and build LAPACK sources with the
-ff2cFortran flag enabled. This is somewhat simpler than compiling e.g. ATLAS.Fixes: gh-2143 (both the BLAS and ARPACK failures), gh-2248, gh-2547
Tested on OSX 10.8.4 (with Accelerate BLAS and FFLAGS set as recommended in the error messages + used to compile LAPACK). Also tested on standard Linux installs.
NOTE: I don't have access to MKL, so I didn't test whether the build still works with it. Anyway, MKL on OSX is suspectible to the same ABI issues and lack of wrapper coverage as Accelerate.
The point that the ABI wrapper coverage could be extended is partly correct, but nobody has stepped up to do this, and we should not keep the master in a broken state wrt. this issue (i.e. that there is in theory also a second solution does not block merging this one IMHO). The ABI checks can be made less strict, but only after the wrappers have sufficient coverage.