Skip to content

Bugfix/matrix buffer protocol#498

Closed
willstott101 wants to merge 3 commits intogazebosim:gz-math7from
willstott101:bugfix/matrix-buffer-protocol
Closed

Bugfix/matrix buffer protocol#498
willstott101 wants to merge 3 commits intogazebosim:gz-math7from
willstott101:bugfix/matrix-buffer-protocol

Conversation

@willstott101
Copy link
Copy Markdown

@willstott101 willstott101 commented Sep 8, 2022

Summary

>>> list(np.array(Matrix4d()))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: iteration over a 0-d array

Now:

>>> m = Matrix4d(0.0, 0.1, 0.2, 0.3, 10,11,12,13,20,21,22,23,30,31,32,33)
0 0.1 0.2 0.3 10 11 12 13 20 21 22 23 30 31 32 33
>>> memoryview(m)[0,0]
0.0
>>> memoryview(m)[2,1]
21.0
>>> m(2,1)
21.0
>>> np.array(m)[2,1]
21.0
>>> np.array(m).tolist()
[[0.0, 0.1, 0.2, 0.3], [10.0, 11.0, 12.0, 13.0], [20.0, 21.0, 22.0, 23.0], [30.0, 31.0, 32.0, 33.0]]

Checklist

  • Signed all commits for DCO
  • Added tests
  • Updated documentation (as needed)
  • Updated migration guide (as needed)
  • Consider updating Python bindings (if the library has them)
  • codecheck passed (See contributing)
  • All tests passed (See test coverage)
  • While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

I haven't signed off yet cause I want feedback on making data non-private - is there another way to access that data from the python header - some sort of friend thing?

@github-actions github-actions bot added the 🌱 garden Ignition Garden label Sep 8, 2022
@willstott101
Copy link
Copy Markdown
Author

Hopefully failing tests here: https://github.com/willstott101/gz-math/actions/runs/3016276863

@willstott101 willstott101 force-pushed the bugfix/matrix-buffer-protocol branch from 0eb7ec0 to e0b8cd1 Compare September 8, 2022 15:39
Copy link
Copy Markdown
Contributor

@adityapande-1995 adityapande-1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi ! Could you please add signoffs to your commits ? https://gazebosim.org/docs/all/contributing#contributing-code

@willstott101
Copy link
Copy Markdown
Author

I mentioned in the original PR text:

I haven't signed off yet cause I want feedback on making data non-private - is there another way to access that data from the python header - some sort of friend thing?

I'm not happy to sign this off yet so I won't sign off the commits.

@willstott101 willstott101 force-pushed the bugfix/matrix-buffer-protocol branch from e0b8cd1 to b793570 Compare September 8, 2022 16:42
@willstott101
Copy link
Copy Markdown
Author

willstott101 commented Sep 8, 2022

Takes a really long time for the CI to roll through, here's latest runs (since they're not allowed to run here by the looks of it?)

Expected failing new tests: https://github.com/willstott101/gz-math/actions/runs/3016796871
After patches to fix those tests: https://github.com/willstott101/gz-math/actions/runs/3020976151

@willstott101 willstott101 force-pushed the bugfix/matrix-buffer-protocol branch 2 times, most recently from a15db79 to 82b51bd Compare September 8, 2022 18:09
@mjcarroll
Copy link
Copy Markdown

since they're not allowed to run here by the looks of it?

Since it was your first time contributing they had to be manually approved, they will run now (and I don't believe you need approval in the future)

@willstott101 willstott101 force-pushed the bugfix/matrix-buffer-protocol branch 5 times, most recently from a2ccd2c to 909c46b Compare September 9, 2022 09:29
@willstott101
Copy link
Copy Markdown
Author

willstott101 commented Sep 9, 2022

Ok finally happy (with everything but making data public).

Resulting buffers are contiguous, index in the same way whether using numpy, memoryview, or Matrix*.__call__.

This means we can go from Matrix -> flat list with () requires contiguous to cast:
memoryview(m).cast('b', (4 * 4 * 8,)).cast('d', (4 * 4,)).tolist()

And matrix to structured list with (does not require contiguous):
memoryview(m).tolist()

@azeey azeey requested a review from mjcarroll October 3, 2022 18:48
@mjcarroll
Copy link
Copy Markdown

Ok finally happy (with everything but making data public).

Did you have a workaround for this yet?

@willstott101
Copy link
Copy Markdown
Author

Nope, I'm not much of a c++ guy, doubt I'll get to digging to be honest - but happy to update if someone has an idea

@willstott101
Copy link
Copy Markdown
Author

I've had a bit of a dig into this, and I don't see any way to make the python binding have access to the private member. It just doesn't seem possible with the way the templates are used.

I don't think making data public is all that bad, would you be tempted to accept this MR with making the data var public?

Signed-off-by: Will Stott <willstott101@gmail.com>
Signed-off-by: Will Stott <willstott101@gmail.com>
Signed-off-by: Will Stott <willstott101@gmail.com>
@willstott101 willstott101 force-pushed the bugfix/matrix-buffer-protocol branch from 02bfd3c to a361471 Compare December 15, 2022 15:42
@willstott101
Copy link
Copy Markdown
Author

I've signed off the commits

@azeey
Copy link
Copy Markdown
Contributor

azeey commented Dec 15, 2022

How about using accessor functions?

@willstott101
Copy link
Copy Markdown
Author

To what end? This PR is about exposing the underlying multi-dimensional array to python. Accessor functions won't enable that.

The buffer protocol is commonly used for converting to/from numpy arrays for instance. I happen to be using it to get the data as a python list, though that could of course be done slowly with accessor functions, exposing the underlying data using the buffer protocol is more flexible, potentially faster in some scenarios and is very common for matrix/array programming in python.

@scpeters
Copy link
Copy Markdown
Member

I've had a bit of a dig into this, and I don't see any way to make the python binding have access to the private member. It just doesn't seem possible with the way the templates are used.

I don't think making data public is all that bad, would you be tempted to accept this MR with making the data var public?

I would prefer adding an accessor method that returns the data pointer, since it is easier to deprecate methods than member variables. I think we've avoided exposing the raw data pointer to help protect from invalid memory access, but it sounds like there's some demand for it, so we could consider add some warnings in the documentation to be careful with the memory access.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 20, 2022

Codecov Report

Merging #498 (a361471) into gz-math7 (b9d69e5) will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           gz-math7     #498   +/-   ##
=========================================
  Coverage     99.70%   99.70%           
=========================================
  Files            77       77           
  Lines          7007     7026   +19     
=========================================
+ Hits           6986     7005   +19     
  Misses           21       21           
Impacted Files Coverage Δ
include/gz/math/Matrix3.hh 100.00% <ø> (ø)
include/gz/math/Matrix4.hh 100.00% <ø> (ø)
include/gz/math/Matrix6.hh 100.00% <ø> (ø)
include/gz/math/detail/AxisIndex.hh 100.00% <0.00%> (ø)
include/gz/math/TimeVaryingVolumetricGrid.hh 100.00% <0.00%> (ø)
...de/gz/math/TimeVaryingVolumetricGridLookupField.hh 100.00% <0.00%> (ø)
include/gz/math/VolumetricGridLookupField.hh 96.51% <0.00%> (+0.12%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@willstott101
Copy link
Copy Markdown
Author

willstott101 commented Dec 20, 2022

I see, I hadn't considered deprecation processes. That makes sense, thank you for explaining.

I understand there might be some hesitance with exposing the underlying array. Definitely rarely need to, particularly when used in a language with very fast (or even inlined) function calls. Sadly python isn't that :(

Would the return type simply be T*? Any appetite for std::array<std::array<T, 3>, 3>& instead? Either using reinterpret_cast, or actually changing the underlying member type.

Perhaps that would help reduce the likelyhood of using it incorrectly. And also make it easier to support a strongly-typed copying getter/setter pair if the reference returning method had to be deprecated.

Either way I'm happy to update this MR with a reference/pointer returning method.

@willstott101
Copy link
Copy Markdown
Author

Closing in favour of #524

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🌱 garden Ignition Garden

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

6 participants