Report targets with falsifying example(s)#2393
Conversation
| else: | ||
| lines = ["Highest target scores:"] | ||
| for label, score in sorted(best_targets.items(), key=lambda x: x[::-1]): | ||
| lines.append("{:>16g} (label={!r})".format(score, label)) |
There was a problem hiding this comment.
Why are we using :>16g for multiple targets and :g for single ones?
There was a problem hiding this comment.
Also would it make sense to indent these lines?
There was a problem hiding this comment.
:>16g right-aligns the numbers, with a four-space indent - see the second example above. I guess we could use :>12g and four spaces if you prefer?
| return [] | ||
| elif len(best_targets) == 1: | ||
| label, score = next(iter(best_targets.items())) | ||
| return ["Highest target score: {:g} (label={!r})".format(score, label)] |
There was a problem hiding this comment.
Nit: Why are we putting the label after the score? The other way around seems like it would make more sense.
There was a problem hiding this comment.
(score, label) matches the signature target(score, *, label="") and lets us reliably align a table when there are several lines of output. For the single-label case I plead consistency (and not that it's often ""), but happy to change this one if you prefer.
Here's an idea that closes #2180 (cc @aarchiba): consider a test which asserts that the largest integer is
100. Obviously this will fail, but shrinking will make it look like an off-by-one error!Fortunately, reporting the highest observation seen by
targetmakes it easy to tell that whilex=101was the minimal counterexample, much larger errors exist:Highest target score: 9.14676e+08 (label='') Falsifying example: test_threshold_problem( x=101, ) Traceback (most recent call last): ...And if we un-comment our other
targetcalls and assertions, we can see that this works smoothly with multi-objective optimisation and reporting of multiple errors too:Highest target scores: 3.27686e+06 (label='') 6.55371e+06 (label='another target') Falsifying example: test_threshold_problem( x=100001, ) Traceback (most recent call last): ... Falsifying example: test_threshold_problem( x=101, ) Traceback (most recent call last): ...As in #2234, either there's a threshold (in which case the maximum is always from a failing test) or there isn't (and the score is irrelevant). For consistency we therefore report the maximum score seen from any example and trust that users can skip over those which are not relevant. While more verbose, I expect this to be a substantial improvement for scientific and data-oriented users where
target()ing an error budget is a very common style of test.