Skip to content

MAINT: gracefully shuffle memoryviews#18282

Merged
charris merged 2 commits intomasterfrom
treddy_issue_18273
Feb 3, 2021
Merged

MAINT: gracefully shuffle memoryviews#18282
charris merged 2 commits intomasterfrom
treddy_issue_18273

Conversation

@tylerjereddy
Copy link
Copy Markdown
Contributor

Fixes gh-18273

  • allow graceful shuffling of memoryviews, with
    same behavior as arrays, instead of producing
    a warning on memoryview shuffle

There's certainly some contention that me doing this downstream was sub-optimal anyway, but I don't think there's any claim that it should be disallowed/considered a bug to shuffle a memoryview.

I was originally going to special case 1-D C-contiguous memoryviews, but the discussion in the matching issue seems to suggest that just using asarray() without fancy special casing might be the way to go?

* allow graceful shuffling of memoryviews, with
same behavior as arrays, instead of producing
a warning on `memoryview` shuffle
char* x_ptr
char* buf_ptr

if isinstance(x, memoryview):
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing this conditional could just be removed? --was being conservative I suppose

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since x can be a list that we definitely do not want to view as an ndarray, this conditional is not optional.

@eric-wieser
Copy link
Copy Markdown
Member

eric-wieser commented Feb 1, 2021

I wonder if we should instead change the MutableSequence check to Sequence? That would allow that path to automatically work for memoryview, and would avoid spurious warnings that precede errors when readonly sequences like tuple.

@charris charris added the 09 - Backport-Candidate PRs tagged should be backported label Feb 1, 2021
@rkern
Copy link
Copy Markdown
Member

rkern commented Feb 1, 2021

There's certainly some contention that me doing this downstream was sub-optimal anyway, but I don't think there's any claim that it should be disallowed/considered a bug to shuffle a memoryview.

That was specific to that particular use of memoryviews post-shuffle, not the shuffling per se. This is a good change.

@seberg
Copy link
Copy Markdown
Member

seberg commented Feb 1, 2021

Both changes seem fine to me (isinstance check or Sequence). Since a non-mutable sequence should error already, I somewhat doubt that modifying that as Eric suggested has any downsides. Presumably, any of the tensor frameworks will either be both or neither. So I think Eric's suggestion is probably good.

I am not quite sure that the memoryview change can't go wrong in principle (memoryviews do support suboffsets for example), but I am willing to take the risk of such an unlikely backcompat break (it would go to an error), it probably is nicer for memoryviews. Although, I doubt there is a reason to optimize for them here.

* based on reviewer feedback, instead of explicitly
supporting `memoryview` shuffling via `asarray()`, support
the shuffling implicitly by using `Sequence` instead
of `MutableSequence`
@tylerjereddy
Copy link
Copy Markdown
Contributor Author

I pushed in Eric's suggestion after confirming the random suite still passes locally for this feature branch.

@seberg
Copy link
Copy Markdown
Member

seberg commented Feb 2, 2021

Thanks, LGTM. I find it a bit surprising that a memoryview isn't considered a mutable sequence (is that supposed to be only the length?), but I guess if Python changes it, we have a test that notifies us.

I would be fine with a memoryview fast path as you had it, although I guess this might be nicer for backporting.

@eric-wieser
Copy link
Copy Markdown
Member

eric-wieser commented Feb 2, 2021

MutableSequences have to implement insert, which memoryview would never do.

@seberg
Copy link
Copy Markdown
Member

seberg commented Feb 2, 2021

Ah right. So I think this is still good enough to warn for tensorflow as well? In which case I think we should go with it or is there some other reason to prefer MutableSequence, that I am missing?

Although, I would be happy to entertain the thought of making things more strict in 1.21.

@seberg seberg added this to the 1.20.1 release milestone Feb 2, 2021
@charris charris merged commit 9f12028 into master Feb 3, 2021
@charris
Copy link
Copy Markdown
Member

charris commented Feb 3, 2021

Thanks Tyler.

@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Feb 4, 2021
@charris charris removed this from the 1.20.1 release milestone Feb 4, 2021
@charris charris deleted the treddy_issue_18273 branch November 3, 2021 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MAINT: shuffle warning specificity

5 participants