Skip to content

Report targets with falsifying example(s)#2393

Merged
Zac-HD merged 1 commit into
HypothesisWorks:masterfrom
Zac-HD:report-targets
Apr 15, 2020
Merged

Report targets with falsifying example(s)#2393
Zac-HD merged 1 commit into
HypothesisWorks:masterfrom
Zac-HD:report-targets

Conversation

@Zac-HD

@Zac-HD Zac-HD commented Apr 13, 2020

Copy link
Copy Markdown
Member

Here's an idea that closes #2180 (cc @aarchiba): consider a test which asserts that the largest integer is 100. Obviously this will fail, but shrinking will make it look like an off-by-one error!

@given(st.integers(min_value=0))
def test_threshold_problem(x):
    target(float(x))
    # target(float(x * 2), label="another target")
    # assert x <= 100000
    assert x <= 100

Fortunately, reporting the highest observation seen by target makes it easy to tell that while x=101 was the minimal counterexample, much larger errors exist:

Highest target score: 9.14676e+08  (label='')

Falsifying example: test_threshold_problem(
    x=101,
)
Traceback (most recent call last): ...

And if we un-comment our other target calls and assertions, we can see that this works smoothly with multi-objective optimisation and reporting of multiple errors too:

Highest target scores:
     3.27686e+06  (label='')
     6.55371e+06  (label='another target')

Falsifying example: test_threshold_problem(
    x=100001,
)
Traceback (most recent call last): ...

Falsifying example: test_threshold_problem(
    x=101,
)
Traceback (most recent call last): ...

As in #2234, either there's a threshold (in which case the maximum is always from a failing test) or there isn't (and the score is irrelevant). For consistency we therefore report the maximum score seen from any example and trust that users can skip over those which are not relevant. While more verbose, I expect this to be a substantial improvement for scientific and data-oriented users where target()ing an error budget is a very common style of test.

@Zac-HD Zac-HD added the legibility make errors helpful and Hypothesis grokable label Apr 13, 2020
else:
lines = ["Highest target scores:"]
for label, score in sorted(best_targets.items(), key=lambda x: x[::-1]):
lines.append("{:>16g} (label={!r})".format(score, label))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using :>16g for multiple targets and :g for single ones?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also would it make sense to indent these lines?

@Zac-HD Zac-HD Apr 14, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:>16g right-aligns the numbers, with a four-space indent - see the second example above. I guess we could use :>12g and four spaces if you prefer?

return []
elif len(best_targets) == 1:
label, score = next(iter(best_targets.items()))
return ["Highest target score: {:g} (label={!r})".format(score, label)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Why are we putting the label after the score? The other way around seems like it would make more sense.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(score, label) matches the signature target(score, *, label="") and lets us reliably align a table when there are several lines of output. For the single-label case I plead consistency (and not that it's often ""), but happy to change this one if you prefer.

@Zac-HD Zac-HD merged commit bd74366 into HypothesisWorks:master Apr 15, 2020
@Zac-HD Zac-HD deleted the report-targets branch July 23, 2020 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

legibility make errors helpful and Hypothesis grokable

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hypothesis should provide assert_almost_equal()

2 participants