Skip to content

BUILD: Remove Accelerate support#15759

Merged
seberg merged 3 commits intonumpy:masterfrom
mattip:remove-accelerate2
May 22, 2020
Merged

BUILD: Remove Accelerate support#15759
seberg merged 3 commits intonumpy:masterfrom
mattip:remove-accelerate2

Conversation

@mattip
Copy link
Copy Markdown
Member

@mattip mattip commented Mar 15, 2020

Redo gh-14880 now that we have a test for Accelerate (gh-15647)

@mattip mattip force-pushed the remove-accelerate2 branch from 72a7890 to 99aad6c Compare March 15, 2020 18:32
@mattip mattip changed the title BUILD: Remove accelerate2 BUILD: Remove Accelerate Mar 15, 2020
@mattip
Copy link
Copy Markdown
Member Author

mattip commented Mar 15, 2020

heh, now the test in gh-15647 seems to fail. Is it building with Accelerate masquerading as openblas or using lapack-lite?

@vrakesh
Copy link
Copy Markdown
Member

vrakesh commented Mar 16, 2020

The test is likely failing, because it is building with lapack_lite. With no explicit accelerate mentioned in the python script it does not search for it. On azure pipeline mac machine unlike my personal mac laptop, the default lapack does not link to accelerate, so the build, for the new test does not build with accelerate, hence the test passes(resulting in pipeline step failure ), since there is not import time issue as far as the pipeline is concerned.

I mentioned the above in
gh-15695

The solution is for the azure pipleine step testing for accelerate error, should be forced built with accelerate for it work on azure pipeline mac instance

Testing the branch on my local Mac continues to detect the issue (since the default lapack is a symlink to accelerate on my mac laptop)

 git branch
  mac_os_test
  master
* remove-accelerate2
(numpyenv) vrakesh$ python -c "import numpy as np"
python(32720,0x10529b5c0) malloc: can't allocate region
*** mach_vm_map(size=18446744072377757696) failed (error code=3)
python(32720,0x10529b5c0) malloc: *** set a breakpoint in malloc_error_break to debug
init_dgelsd failed init
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/rakvas/Workspace/numpy/numpy/__init__.py", line 287, in <module>
    raise RuntimeError(msg)
RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

(numpyenv) vrakesh$ 

@mattip
Copy link
Copy Markdown
Member Author

mattip commented Mar 17, 2020

I wonder why your machine has symlinks and the azure build machine does not. Did Azure remove something or did you (and @WarrenWeckesser, who also noticed the symlinks) somehow install something that created those links? Versions of Xcode?

@mattip
Copy link
Copy Markdown
Member Author

mattip commented Mar 17, 2020

On the other hand, maybe it is enough that gh-15695 showed that the test works, and now we can remove that CI run confident that anyone who still uses Accelerate via those symlinks will get the error.

@vrakesh
Copy link
Copy Markdown
Member

vrakesh commented Mar 17, 2020

I wonder why your machine has symlinks and the azure build machine does not. Did Azure remove something or did you (and @WarrenWeckesser, who also noticed the symlinks) somehow install something that created those links? Versions of Xcode?

I am not sure why either, but I ran a script on azure pipeline, to verify it. there is no symlink back to accelerate for the default lapack and blas implementation.

I am fine with disabling the test as well on pipelines or we could force the symlink ourselves, in the pipeline script (Though I would not prefer the second option, since it disturbs the preset environment)

@mattip mattip requested a review from WarrenWeckesser March 18, 2020 11:11
Copy link
Copy Markdown
Member

@cournape cournape left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this cruft makes me so happy. I still have scars from the time where we looked into debugging accelerate :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we don't simply remove accelerate_info ? From a backward compatibility perspective, I don't think it would make sense to continue using accelerate_info in any other package ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There used to be quite a few packages using blas_opt or lapack_opt. Accelerate is actually less buggy and easier to use that picking a random version of OpenBLAS, hence keeping accelerate_info for a while longer could make sense.

That said, a code search of the likes of Statsmodels and PyAMG shows that they've stopped using *_opt, so the impact of doing it now is likely to be very limited.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be happy to remove it, but then we might need to do a 19.1 to revert the change. Perhaps it should be a separate PR to make reverting easier?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as above ? Shouldn't we remove it completely and then remove those specific tests ? Or is there some subtle interaction I am missing ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattip can you briefly comment on this? Should this be an error, or a warning?

Anyway, I would like to put this in pretty much as is (just a general note for those not at the triage meeting).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding a warning here might be too noisy: it would trigger whenever a build is done on macOS. Once this PR is in, we had discussed totally removing Accelerate discovery from the code as a single PR that can be reverted if it breaks downstream packages. Note the code here is NumPy specific and does not affect downstream.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to keep this commented code ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not sure how the change would be received, so preferred to comment it out rather than remove for easy reversal.

Copy link
Copy Markdown
Member

@vrakesh vrakesh Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree keeping this commented, for now makes sense. If we do not see any issues from broader community, we can remove it always.

@mattip
Copy link
Copy Markdown
Member Author

mattip commented Mar 30, 2020

@cournape do you still have questions? The GUI says "changes requested" but I think I related to all the review comments.

@mattip mattip force-pushed the remove-accelerate2 branch from d16cbe4 to 9fdf2ec Compare April 7, 2020 10:44
@mattip
Copy link
Copy Markdown
Member Author

mattip commented Apr 7, 2020

I have addressed @cournape 's comments. It would be nice to get this in before it once again goes stale and needs a rebase.

@mattip mattip added the triage review Issue/PR to be discussed at the next triage meeting label Apr 8, 2020
@charris
Copy link
Copy Markdown
Member

charris commented Apr 9, 2020

So this PR should wait until the test is improved, and then we can revisit removing the Accelerate-specific code.

Yes, the idea is that we do not want accelerate to be used either by intent or by accident.

@WarrenWeckesser
Copy link
Copy Markdown
Member

Rather than repeat it here, I'll link to my comment on gh-15946.

@mattip mattip removed the triage review Issue/PR to be discussed at the next triage meeting label Apr 22, 2020
@WarrenWeckesser
Copy link
Copy Markdown
Member

Instead of continuing #15966, it seems simpler to add the simple check for a symlink to Accelerate to this PR. I created a branch in my repo that has one more commit on top of this PR: WarrenWeckesser@ac55699

I think that does what we want.

MAINT: distutils: Error when a symlink to Accelerate is found.
@seberg
Copy link
Copy Markdown
Member

seberg commented May 22, 2020

Thanks Matti and Warren! There seem to be some unaligned dot tests around, so I think fully deleting the test is just as well. Lets put this in, if anything comes up, we can fix it up.

@seberg seberg merged commit 78593a1 into numpy:master May 22, 2020
@seberg seberg changed the title BUILD: Remove Accelerate BUILD: Remove Accelerate support May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants