Skip to content

Fixes for changes in how object arrays are created in numpy-dev#9719

Merged
mhvk merged 4 commits intoastropy:masterfrom
mhvk:numpy-dev-object-array-changes
Jan 22, 2020
Merged

Fixes for changes in how object arrays are created in numpy-dev#9719
mhvk merged 4 commits intoastropy:masterfrom
mhvk:numpy-dev-object-array-changes

Conversation

@mhvk
Copy link
Contributor

@mhvk mhvk commented Dec 3, 2019

Fixes #9716. Do need to check numpy-dev travis run before merging!! Locally, I saw some failures still, but those are not related to this PR, but rather to #9672 (would be good to merge that...) EDIT: rebased after #9762 was merged: now should give an all-green numpy-dev.

The changes for coordinates.Angle (cc @astrofrog) and table (@taldcroft) are trivial (just adjustments to how the test is done), but that to time a bit less so, as it involves more than wanted special-casing of a list of times. My sense would be that it would be better to move this check earlier, before the general _init_from_vals is run, but that is somewhat out of scope here. @taldcroft, what do you think?

EDIT: Fix #9879

assert a6.unit is u.degree

with pytest.raises(TypeError):
with ExitStack() as stack:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just curious, I have seen ExitStack being added in this PR and another completely unrelated PR. Is this the new way to do pytest.raises or just context manager in general?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is no coincidence that you see it suddenly in two places - I saw that completely unrelated PR as well, and thought that it was a neat way to combine context managers, especially if, as here, the number of context managers included is not always the same (without it, I would have needed to use the "null context", which I don't really like - though better than using ExitStack when one has either one or zero context managers).

Copy link
Member

@astrofrog astrofrog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

if not NUMPY_LT_1_18:
stack.enter_context(
pytest.warns(DeprecationWarning,
match='automatic object dtype is deprecated'))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we aim to make sure the code below doesn't emit a warning?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait that's due to numpy/numpy#15041 right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, numpy emits the warning. Since this is just a test that something fails, I think it is not all that important whether there is a warning as well. Eventually, the warning might just change to another exception, but that is OK too.

@mhvk
Copy link
Contributor Author

mhvk commented Dec 4, 2019

Great, all tests now pass. @taldcroft, if you have time, could you have a look at the change in Time? If not, I think it is OK to merge this now just so that things work with numpy-dev, and then I'll raise a new issue to get us thinking about how to handle input consisting of lists of Time instances (e.g., part of TimeAstropyTime seems written with the idea that one could even enter multi-D nested lists - passing through np.block might make this work!)

@mhvk
Copy link
Contributor Author

mhvk commented Dec 5, 2019

Hmm, the changes are being reverted for numpy 1.18 (though they'll likely stay in numpy-dev). That gives some time to think this through for us too.

else:
dtype = None

val = np.array(val, copy=copy, subok=True, dtype=dtype)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this instead with a try/except/catch-warning? Basically try doing val = np.array(val, copy=copy, subok=True) and then if numpy emits a warning or raises an exception then fall through to explicitly specifying dtype=object. That would avoid the special case and hopefully just delegate to numpy whether or not the original val is acceptable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@taldcroft - coming back to this PR after a while, I remember why I did the upfront test: I'm quite happy to just do try/except for the exception, but changing the warnings filter seems slow and excessive. So, the question then is whether we mind the warnings for the case of [scalar_time, array_time].

p.s. Ideally, to me, the case of an iterable of time would never get turned into an array in the first place...

Copy link
Member

@taldcroft taldcroft Jan 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a little profiling, and the impact on initialization of catching warnings is not huge, probably an increase of 2 us on a typical initialization of 65 us. The catching bit could be removed once versions of numpy that issue the deprecation are no longer supported by astropy.

Overall I agree that the current API of allowing a list of ragged Time objects and concatenating is odd and in fact I was surprised myself at that behavior. 😄. But that's a separate decision about whether to deprecate that as well.

I think that allowing a (non-ragged) iterable of Time is OK, certainly given the current behavior:

>>> t = Time([1,2,3], format='cxcsec')
>>> np.array(t)
array([<Time object: scale='tt' format='cxcsec' value=1.0>,
       <Time object: scale='tt' format='cxcsec' value=2.0>,
       <Time object: scale='tt' format='cxcsec' value=3.0>], dtype=object)
>>> Time(np.array(t))
<Time object: scale='tt' format='cxcsec' value=[1. 2. 3.]> 
# Round trips!  

Timing

import warnings
from astropy.time import Time
def fxn(**kwargs):
    x = np.array([1], **kwargs)

def do_fxn():
    with warnings.catch_warnings(record=True) as w:
        try:
            fxn()
        except TypeError:
            fxn(dtype=object)
            
        if len(w) > 0:
            # Check for the automatic object array deprecation warning and ignore,
            # otherwise emit the warning.
            pass
timeit do_fxn()
2.73 µs ± 31.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
timeit fxn()
667 ns ± 14.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
timeit Time(50000, format='mjd')
63 µs ± 1.9 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
timeit Time('2019-01-01', format='iso')
66.5 µs ± 1.37 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might work, though it means the type of any other warning is lost (only keeps the warning message), and we add yet more that is not thread-safe. The logical route here would be not to record but set up an appropriate filter instead, but that adds another few us, while my if statement is just one line and well under a micro-second.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, fair enough. I do think that this kind of special-case opt-in catching is fragile and has some risk of failing for an oddball case you haven't thought about. But I can't decide if that is worse or doing just try/except and exposing annoying warnings that will be obscure to the user. So I'll let you decide. 😄

@pllim
Copy link
Member

pllim commented Dec 12, 2019

I don't see the failure on CI anymore. Is this still an issue?

@pllim pllim modified the milestones: v4.0.1, v4.1 Dec 12, 2019
@bsipocz
Copy link
Member

bsipocz commented Dec 13, 2019

@pllim - what about 1.18rc? Has this been revoked from there, too?

@pllim
Copy link
Member

pllim commented Dec 13, 2019

what about 1.18rc?

@mhvk would know? 🤷‍♀

@bsipocz
Copy link
Member

bsipocz commented Jan 21, 2020

@mhvk - could you please rebase? (A lot has changed both here and upstream, would be nice to get a new CI run)

@mhvk mhvk force-pushed the numpy-dev-object-array-changes branch from ab8f06c to 7d9a378 Compare January 21, 2020 22:03
@mhvk
Copy link
Contributor Author

mhvk commented Jan 21, 2020

Rebased, but didn't check locally that the changes are still correct, so definitely need to look at the numpy-dev result!

Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

numpy-dev job is all green, so I would love to go ahead and merged this!

# Arrays where the elements are Angle objects are not supported -- it's
# really tricky to do correctly, if at all, due to the possibility of
# nesting.
if not NUMPY_LT_1_18:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this may need to be NUMPY_LT_1_19 now, but since everything passed, including the np 1.18 job, maybe it's still OK like this?

@mhvk
Copy link
Contributor Author

mhvk commented Jan 21, 2020

@bsipocz - looking through, I see there still is a comment by @taldcroft that I didn't address. I'll try to do so later today.

I am puzzled about the NUMPY_LT_1_18 vs *_19: I thought that in numpy 1.18 no warning would be raised. Will look into it a little.

@mhvk
Copy link
Contributor Author

mhvk commented Jan 22, 2020

I changed to NUMPY_LT_1_19 (not completely sure why NUMPY_LT_1_18 worked on 1.18, since that doesn't emit a warning, but o well), but right now left the code as it is for time. My suggestion would be to merge as is, and open an issue about the comment @taldcroft made - I'd like to prevent lists/tuples of Time instances even getting to the stage where they are causing problems.

@pllim
Copy link
Member

pllim commented Jan 22, 2020

My suggestion would be to merge as is, and open an issue

@taldcroft , what say you?

@mhvk
Copy link
Contributor Author

mhvk commented Jan 22, 2020

OK, let's merge this. See #9882 (getting close to 10000...) to remind us of the question of how to do things in Time.

@mhvk mhvk merged commit 0d833cc into astropy:master Jan 22, 2020
@mhvk mhvk deleted the numpy-dev-object-array-changes branch January 22, 2020 20:27
bsipocz pushed a commit that referenced this pull request Jan 22, 2020
Fixes for changes in how object arrays are created in numpy-dev
bsipocz pushed a commit that referenced this pull request Jan 22, 2020
Fixes for changes in how object arrays are created in numpy-dev
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TST: VisibleDeprecationWarning causing numpy-dev failures Test failures with Numpy dev due to deprecation warning related to object dtype

5 participants