Skip to content

Fix maybe_promote#1953

Merged
shoyer merged 4 commits intopydata:masterfrom
NotSqrt:1952-fix-maybe-promote
Aug 20, 2018
Merged

Fix maybe_promote#1953
shoyer merged 4 commits intopydata:masterfrom
NotSqrt:1952-fix-maybe-promote

Conversation

@NotSqrt
Copy link
Copy Markdown
Contributor

@NotSqrt NotSqrt commented Mar 2, 2018

With tests for every possible dtype:

(numpy docs say biufcmMOSUV only)

for letter in string.ascii_letters:
    try:
        print(letter, np.dtype(letter))
    except TypeError as exc:
        pass
  • Closes xarray.merge exception : invalid type promotion #1952
  • Tests added (for all bug fixes or enhancements)
  • Tests passed (for all non-documentation changes)
  • Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

With tests for every possible dtype:

(numpy docs say `biufcmMOSUV` only)

```
for letter in string.ascii_letters:
    try:
        print(letter, np.dtype(letter))
    except TypeError as exc:
        pass
```
Copy link
Copy Markdown
Member

@shoyer shoyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together so quickly, and for writing tests for this function!

"""
# N.B. these casting rules should match pandas
if np.issubdtype(dtype, np.floating):
if np.issubdtype(dtype, np.timedelta64):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment with a link to the numpy bug explaining why we need to do this first?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe move this below np.floating -- there was an intention to hit more commonly used dtypes first.

actual = dtypes.maybe_promote(np.dtype(kind))
if np.issubdtype(np.dtype(kind), np.complexfloating) or kind in 'mM':
assert actual[0] == expected[0]
assert str(actual[1]) == str(expected[1])
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of keeping things as tuples, can we split actual/expected into values and dtypes? This would make things a little clearer:

assert actual_dtype == expected_dtype
assert str(actual_value) == str(expected_value)

assert actual[0] == expected[0]
assert str(actual[1]) == str(expected[1])
else:
assert actual == expected
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably better not to rely on this code-path ever working -- it relies on quirks of tuples using object identity in comparisons in a way that is arguably broken for NaN.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I skipped all complex dtypes.

('F', (np.complex64, np.complex('nan+nanj'))), # dtype('complex64')
('f', (np.float32, np.nan)), # dtype('float32')
('G', (np.complex256, np.complex('nan+nanj'))), # dtype('complex256')
('g', (np.float128, np.nan)), # dtype('float128')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently complex256 and float128 aren't always available -- that's why our integration tests on Appveyor (running Windows) fail for this PR. So let's exclude these.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ignored float128 (and complex256, see #1953 (comment))

@shoyer
Copy link
Copy Markdown
Member

shoyer commented Mar 9, 2018

@NotSqrt this is really close -- any interest in finishing this up? :)

@NotSqrt
Copy link
Copy Markdown
Contributor Author

NotSqrt commented Aug 16, 2018

@shoyer Can you review those changes, please ? Thanks !

@shoyer shoyer merged commit 69086b3 into pydata:master Aug 20, 2018
@shoyer
Copy link
Copy Markdown
Member

shoyer commented Aug 20, 2018

thanks @NotSqrt

@NotSqrt NotSqrt deleted the 1952-fix-maybe-promote branch August 20, 2018 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

xarray.merge exception : invalid type promotion

2 participants