Implement Python Array API asarray function.#60627
Implement Python Array API asarray function.#60627ysiraichi wants to merge 25 commits intogh/ysiraichi/13/basefrom
asarray function.#60627Conversation
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
🔗 Helpful links
💊 CI failures summary and remediationsAs of commit e1e6115 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 60b13a7 Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 681ab0d Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 8b8b22c Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 5ae565c Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 3bd82ef Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 78bce4f Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 20b3ced Pull Request resolved: #60627
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 71f40cb Pull Request resolved: #60627
|
Hi, @anjali411 @mruberry. This is a friendly reminder about this PR. Do you have some time to take a look at it? |
| >>> a = torch.tensor([1, 2, 3]) | ||
| >>> # Shares memory with tensor 'a' | ||
| >>> b = torch.asarray(a) | ||
| >>> a.data_ptr() == b.data_ptr() |
There was a problem hiding this comment.
okay yes so based on this, we should disable asarray for conjugated and negated tensors (see the comment above pointing to a PR that made a similar change for .numpy())
| def get_dtype_size(dtype): | ||
| return int(torch.empty((), dtype=dtype).element_size()) | ||
|
|
||
| class TestBufferProtocol(TestCase): |
There was a problem hiding this comment.
was this just verbatim moved from the file deleted above?
|
Hi @anjali411. Thank you for your review.
I will do it in another PR. 😄 |
|
Hi, @mruberry. This is a friendly reminder about this PR. Do you have some time to take a look at it? |
| bool wrong_device = device.has_value() && device.value() != tensor.device(); | ||
| bool wrong_dtype = | ||
| dtype.has_value() && dtype.value() != tensor.scalar_type(); | ||
| bool needs_copying = !copy.has_value() && (wrong_device || wrong_dtype); |
There was a problem hiding this comment.
This was a tricky conditional to read because needs_copying would seem to be determined by whether the device or dtype is incorrect but the additional !copy.has_value() is confusing. Not worth revising the PR for
There was a problem hiding this comment.
Ah. You are totally right.
mruberry
left a comment
There was a problem hiding this comment.
Hey @ysiraichi!
Thank you for your patience and pings. I finally had a chance to read through this PR again thoroughly (I tried several times but something always came up!). It's very good, and I appreciate @anjali411's concern but don't think this PR suffers from the issue. There are a few small tweaks I would like to suggest about readability and docs, but they're minor and I think we should land this as-is. Nice work, this is a challenging, detailed PR that required a great deal of effort, thoughtfulness, and attention to detail.
|
@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@mruberry @anjali411 Thank you for the review! I imagine it was a little bit tricky to review it, since I moved some code around in some non-straightforward way. Anyway, thank you for reviewing it. |
|
Ack! Looks like this just needs a rebase, @ysiraichi. Sorry about that. Just ping me once it's rebased. |
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray Differential Revision: [D31640510](https://our.internmc.facebook.com/intern/diff/D31640510) [ghstack-poisoned]
CI Flow Status⚛️ CI FlowRuleset - Version:
You can add a comment to the PR and tag @pytorchbot with the following commands: # ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun
# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slowFor more information, please take a look at the CI Flow Wiki. |
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 4816662 Pull Request resolved: #60627
|
@mruberry Rebased! |
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray Differential Revision: [D31640510](https://our.internmc.facebook.com/intern/diff/D31640510) [ghstack-poisoned]
In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray ghstack-source-id: 796f54f Pull Request resolved: #60627
CI Flow Status⚛️ CI FlowRuleset - Version:
You can add a comment to the PR and tag @pytorchbot with the following commands: # ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun
# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slowFor more information, please take a look at the CI Flow Wiki. |
1 similar comment
CI Flow Status⚛️ CI FlowRuleset - Version:
You can add a comment to the PR and tag @pytorchbot with the following commands: # ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun
# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slowFor more information, please take a look at the CI Flow Wiki. |
|
@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Summary: Pull Request resolved: #60627 In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray Test Plan: Imported from OSS Reviewed By: H-Huang Differential Revision: D31640510 Pulled By: mruberry fbshipit-source-id: d0869e0d73cb50023d5866b001dac5d34ca30dfd
Summary: Pull Request resolved: pytorch#60627 In this PR, the core of `frombuffer` and `fromDLPack` onto _tensor_new.cpp_. `asarray` uses such refactored functions for interpreting the object as a tensor. We follow the Python Array API standard found: https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray Test Plan: Imported from OSS Reviewed By: H-Huang Differential Revision: D31640510 Pulled By: mruberry fbshipit-source-id: d0869e0d73cb50023d5866b001dac5d34ca30dfd
Stack from ghstack:
asarrayfunction. #60627In this PR, the core of
frombufferandfromDLPackonto tensor_new.cpp.asarrayuses such refactored functions for interpreting the object as a tensor. We follow the
Python Array API standard found:
https://data-apis.org/array-api/latest/API_specification/creation_functions.html?highlight=asarray
Differential Revision: D31640510