BUILD: Remove Accelerate support#15759
Conversation
72a7890 to
99aad6c
Compare
|
heh, now the test in gh-15647 seems to fail. Is it building with Accelerate masquerading as openblas or using lapack-lite? |
|
The test is likely failing, because it is building with lapack_lite. With no explicit accelerate mentioned in the python script it does not search for it. On azure pipeline mac machine unlike my personal mac laptop, the default lapack does not link to accelerate, so the build, for the new test does not build with accelerate, hence the test passes(resulting in pipeline step failure ), since there is not import time issue as far as the pipeline is concerned. I mentioned the above in The solution is for the azure pipleine step testing for accelerate error, should be forced built with accelerate for it work on azure pipeline mac instance Testing the branch on my local Mac continues to detect the issue (since the default lapack is a symlink to accelerate on my mac laptop) git branch
mac_os_test
master
* remove-accelerate2
(numpyenv) vrakesh$ python -c "import numpy as np"
python(32720,0x10529b5c0) malloc: can't allocate region
*** mach_vm_map(size=18446744072377757696) failed (error code=3)
python(32720,0x10529b5c0) malloc: *** set a breakpoint in malloc_error_break to debug
init_dgelsd failed init
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/rakvas/Workspace/numpy/numpy/__init__.py", line 287, in <module>
raise RuntimeError(msg)
RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned
(numpyenv) vrakesh$ |
|
I wonder why your machine has symlinks and the azure build machine does not. Did Azure remove something or did you (and @WarrenWeckesser, who also noticed the symlinks) somehow install something that created those links? Versions of Xcode? |
|
On the other hand, maybe it is enough that gh-15695 showed that the test works, and now we can remove that CI run confident that anyone who still uses Accelerate via those symlinks will get the error. |
I am not sure why either, but I ran a script on azure pipeline, to verify it. there is no symlink back to accelerate for the default lapack and blas implementation. I am fine with disabling the test as well on pipelines or we could force the symlink ourselves, in the pipeline script (Though I would not prefer the second option, since it disturbs the preset environment) |
cournape
left a comment
There was a problem hiding this comment.
Removing this cruft makes me so happy. I still have scars from the time where we looked into debugging accelerate :)
numpy/core/setup.py
Outdated
There was a problem hiding this comment.
Is there a reason we don't simply remove accelerate_info ? From a backward compatibility perspective, I don't think it would make sense to continue using accelerate_info in any other package ?
There was a problem hiding this comment.
There used to be quite a few packages using blas_opt or lapack_opt. Accelerate is actually less buggy and easier to use that picking a random version of OpenBLAS, hence keeping accelerate_info for a while longer could make sense.
That said, a code search of the likes of Statsmodels and PyAMG shows that they've stopped using *_opt, so the impact of doing it now is likely to be very limited.
There was a problem hiding this comment.
I would be happy to remove it, but then we might need to do a 19.1 to revert the change. Perhaps it should be a separate PR to make reverting easier?
numpy/linalg/setup.py
Outdated
There was a problem hiding this comment.
Same question as above ? Shouldn't we remove it completely and then remove those specific tests ? Or is there some subtle interaction I am missing ?
There was a problem hiding this comment.
@mattip can you briefly comment on this? Should this be an error, or a warning?
Anyway, I would like to put this in pretty much as is (just a general note for those not at the triage meeting).
There was a problem hiding this comment.
I think adding a warning here might be too noisy: it would trigger whenever a build is done on macOS. Once this PR is in, we had discussed totally removing Accelerate discovery from the code as a single PR that can be reverted if it breaks downstream packages. Note the code here is NumPy specific and does not affect downstream.
azure-pipelines.yml
Outdated
There was a problem hiding this comment.
Do we need to keep this commented code ?
There was a problem hiding this comment.
I was not sure how the change would be received, so preferred to comment it out rather than remove for easy reversal.
There was a problem hiding this comment.
I agree keeping this commented, for now makes sense. If we do not see any issues from broader community, we can remove it always.
|
@cournape do you still have questions? The GUI says "changes requested" but I think I related to all the review comments. |
d16cbe4 to
9fdf2ec
Compare
|
I have addressed @cournape 's comments. It would be nice to get this in before it once again goes stale and needs a rebase. |
Yes, the idea is that we do not want accelerate to be used either by intent or by accident. |
|
Rather than repeat it here, I'll link to my comment on gh-15946. |
|
Instead of continuing #15966, it seems simpler to add the simple check for a symlink to Accelerate to this PR. I created a branch in my repo that has one more commit on top of this PR: WarrenWeckesser@ac55699 I think that does what we want. |
MAINT: distutils: Error when a symlink to Accelerate is found.
|
Thanks Matti and Warren! There seem to be some unaligned dot tests around, so I think fully deleting the test is just as well. Lets put this in, if anything comes up, we can fix it up. |
Redo gh-14880 now that we have a test for Accelerate (gh-15647)