Skip to content

Hypothesis tests for astropy.units#9211

Closed
Zac-HD wants to merge 1 commit into
astropy:masterfrom
Zac-HD:add-hypothesis
Closed

Hypothesis tests for astropy.units#9211
Zac-HD wants to merge 1 commit into
astropy:masterfrom
Zac-HD:add-hypothesis

Conversation

@Zac-HD

@Zac-HD Zac-HD commented Sep 6, 2019

Copy link
Copy Markdown
Contributor

This PR adds Hypothesis tests for astropy.units, as proposed in #9017.

The test will reliably find floating-point imprecision in the scale factors of various units, which can build up over the course of round-trip conversions. I would propose that all units should be defined with the scale factor from fundamental SI units stored as a Fraction, so that conversions can be lossless - the overhead is very small as it is a single number in the metadata for an array of any size.

I've done my best to list the test-time dependency on hypothesis in the right place, but would be very happy if anyone had suggestions or feedback on this 😄

@astrofrog

Copy link
Copy Markdown
Member

@Zac-HD - thanks for the PR! :)

@mhvk

mhvk commented Sep 6, 2019

Copy link
Copy Markdown
Contributor

@Zac-HD - we'd definitely need to run some benchmarks before turning SI prefixes into Fraction - people do use scalar Quantity a lot and we've tried hard to optimize unit operations (e.g., for powers, always using float if a fraction is guaranteed to be exactly representable).

Here, remember that units are almost exclusively used as attributes to Quantity. Since in, e.g., a quantity conversion of say ms to s the fraction of 1/1000 needs to be converted to a float 0.001 anyway in order to multiply it with the underlying float value, I'm not sure that it helps all that much to keep a Fraction around longer. (Plus, Fraction are not much use for conversions to items like M_sun, AU, etc.)

@aarchiba aarchiba mentioned this pull request Sep 20, 2019
@aarchiba

Copy link
Copy Markdown
Contributor

The numerics of units are pretty simple, but parsing Unicode is a classic place to get things disastrously wrong. Perhaps #9348 would be a good place to apply hypothesis?

@bsipocz bsipocz added this to the v4.1 milestone Oct 21, 2019
@Zac-HD

Zac-HD commented Oct 22, 2019

Copy link
Copy Markdown
Contributor Author

Looking at past issues, I think a similar test might have caught #8260, if you could state exact properties (even if only for units that do not involve floating-point factors).

@mhvk

mhvk commented Oct 22, 2019

Copy link
Copy Markdown
Contributor

@Zac-HD - I'd be quite keen to get some form of hypothesis testing in, and this seems a nice one. I think the main question is to ensure this is not run all the time cc @astropy/coordinators

@Zac-HD

Zac-HD commented Oct 22, 2019

Copy link
Copy Markdown
Contributor Author

Why not run it all the time? The performance overhead is tiny, and Hypothesis tests often find bugs that example-based tests don't... but only if they're executed of course!

@mhvk

mhvk commented Oct 22, 2019

Copy link
Copy Markdown
Contributor

Will it still be tiny if we do it for every package? But if it is small compared to the other tests, then it is OK. (Sorry, I'm a bit of a proselitizer for reducing power consumption...)

@bsipocz

bsipocz commented Oct 22, 2019

Copy link
Copy Markdown
Member

I've milestoned it to 4.1 as we plan to have a discussion and possibly sprint around this at the coordination meeting in December. The overall feeling seems to be that we'll need this, but on the other hand this week is feature freeze, so much of the infrastructural changes and refactors are put on the back burner.

@Zac-HD

Zac-HD commented Oct 23, 2019

Copy link
Copy Markdown
Contributor Author

Will [test runtime] still be tiny if we do it for every package? But if it is small compared to the other tests, then it is OK.

I would expect that test runtime would still mostly be example-based rather than property-based tests.

If PBT is taking too long, you can e.g. turn down the max_examples setting to reduce the number of examples each test checks - either globally or just for the slow tests. Do leave it at the defaults unless you actually observe a problem though, we've tuned the time/bugs tradeoff pretty carefully.

I've milestoned it to 4.1 as we plan to have a discussion and possibly sprint around this at the coordination meeting in December. The overall feeling seems to be that we'll need this

🎉

If it would be helpful, I'd be delighted to attend any sessions on testing remotely from Australia to resource the conversation.

@bsipocz

bsipocz commented Oct 23, 2019

Copy link
Copy Markdown
Member

That sounds very useful, thank you. I've added a note to the meeting notes to schedule the discussion that falls into daytime in Canberra (https://docs.google.com/document/d/1pO0giLKGEle5rJeyxhc33rIARY_e_naLiYprScROP84/edit#)

@aarchiba

aarchiba commented Nov 5, 2019

Copy link
Copy Markdown
Contributor

I started applying hypothesis to Time objects in #9532 and have found and filed #9533 .

aarchiba added a commit to aarchiba/astropy that referenced this pull request Nov 6, 2019
If hypothesis ends up fully integrated it should probably be put into
pytest-astropy as a dependency, and some of this config should move
there.
aarchiba added a commit to aarchiba/astropy that referenced this pull request Nov 26, 2019
Import hypothesis setup from PR astropy#9211 - If hypothesis ends up fully
integrated it should probably be put into pytest-astropy as a
dependency, and some of this config should move there.

Introduce non-hypothesis example for use in astropy#9579
aarchiba added a commit to aarchiba/astropy that referenced this pull request Nov 29, 2019
Import hypothesis setup from PR astropy#9211 - If hypothesis ends up fully
integrated it should probably be put into pytest-astropy as a
dependency, and some of this config should move there.

Introduce non-hypothesis example for use in astropy#9579
@Zac-HD

Zac-HD commented Dec 9, 2019

Copy link
Copy Markdown
Contributor Author

@bsipocz @astrofrog (@aarchiba) I have a thing in my calendar to phone in to the coordination meeting if you're discussing Hypothesis. Is there a specific time attached to that now?

@pllim

pllim commented Dec 9, 2019

Copy link
Copy Markdown
Member

@Zac-HD , are you able to join http://joinslack.astropy.org/ ?

@Zac-HD

Zac-HD commented Dec 9, 2019

Copy link
Copy Markdown
Contributor Author

Yep, I'm in. Taking it over there.

@Zac-HD

Zac-HD commented Dec 25, 2019

Copy link
Copy Markdown
Contributor Author

Closing this PR because I think @aarchiba's work is much more convincing about the value of property-based testing for astropy (and astronomy), and the decision to use Hypothesis has been made - it's now provided when you install pytest-astropy.

Thanks to everyone who has been part of these conversations over the last months!

@Zac-HD Zac-HD closed this Dec 25, 2019
@mhvk

mhvk commented Dec 25, 2019

Copy link
Copy Markdown
Contributor

Thank you, @Zac-HD! It will get to units too...

@bsipocz bsipocz removed this from the v4.1 milestone Jan 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants