BENCH: add ufunc argument parsing benchmarks.#11453
Conversation
|
LGTM, but I'm not clear on what "# On purpose pick no-ops" means in all the cases, so could you make the comment more informative? |
058b437 to
8157ba9
Compare
|
OK, I wrote a more general comment for the whole class on the choice of arguments. |
eric-wieser
left a comment
There was a problem hiding this comment.
I'd maybe be tempted to write something weirder like
class CallArgParsing(Benchmark):
param_names = ['args']
params = [
((x, y), {}),
((x, y, out), {}),
((x, y, (out,)), {}),
]
def time_call(self, arg_pair):
np.add(*arg_pair[0], **arg_pair[1])
This has the advantage of not incurring a variable number of self.attribute lookups in each test. It also means the tests will appear on the same graph, since they go through one function.
You'd need a separate benchmark group for reduce, but that's probably fine
|
That sounds like a nice idea. (I wish |
|
How does it not work? |
|
@pv - checking on my office machine, it actually does work. The difference is that there I have python2.7 development stuff installed, while on my laptop, where I tried earlier, I have not. I'll look into where it decides to try python2.7 anyway. |
|
OK, well, that was simple, A separate issue, though: #11458 |
|
The default configuration in the file |
|
Indeed, works for |
|
|
8157ba9 to
2fa3789
Compare
|
@eric-wieser - I updated the benchmarks per your suggestion; certainly the output of just running p.s. It also looks like there is a substantial regression in speed compared to 1.14.4 - much more than what my PRs are improving. Sigh. |
tylerjereddy
left a comment
There was a problem hiding this comment.
I checked out the PR branch and ran the benchmarks locally with asv run -e -b "ArgParsing*" >& log.txt and the output looks sensible (using conda env & Python 2.7 + 3.6):
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py2.7-six
·· Building 5b672cbe for conda-py2.7-six
·· Installing into conda-py2.7-six
· Running 4 total benchmarks (1 commits * 2 environments * 2 benchmarks)
[ 0.00%] · For numpy commit hash 5b672cbe:
[ 0.00%] ·· Building for conda-py2.7-six
[ 0.00%] ·· Benchmarking conda-py2.7-six
[ 25.00%] ··· Running bench_ufunc.ArgParsing.time_add_arg_parsing ok
[ 25.00%] ····
============================================================================ =============
arg_kwarg
---------------------------------------------------------------------------- -------------
((array(1.), array(2.)), {}) 974±3ns
((array(1.), array(2.), array(3.)), {}) 1.00±0.01μs
((array(1.), array(2.)), {'out': array(3.)}) 1.08±0.01μs
((array(1.), array(2.)), {'out': (array(3.),)}) 952±10ns
((array(1.), array(2.)), {'subok': True, 'where': True, 'out': array(3.)}) 1.00±0.01μs
((array(1.), array(2.)), {'subok': True}) 1.12±0.02μs
((array(1.), array(2.)), {'subok': True, 'where': True}) 1.09±0.01μs
((array(1.), array(2.), array(3.)), {'subok': True, 'where': True}) 1.04±0.01μs
============================================================================ =============
[ 50.00%] ··· Running ...ArgParsingReduce.time_add_reduce_arg_parsing ok
[ 50.00%] ····
==================================================================== =============
arg_kwarg
-------------------------------------------------------------------- -------------
((array([0., 1.]),), {}) 1.61±0.02μs
((array([0., 1.]), 0), {}) 1.69±0.01μs
((array([0., 1.]),), {'axis': 0}) 1.98±0.02μs
((array([0., 1.]), 0, None), {}) 1.90±0.03μs
((array([0., 1.]),), {'dtype': None, 'axis': 0}) 1.88±0.01μs
((array([0., 1.]), 0, None, array(0.)), {}) 1.62±0.02μs
((array([0., 1.]),), {'dtype': None, 'out': array(0.), 'axis': 0}) 1.81±0.01μs
((array([0., 1.]),), {'out': array(0.)}) 1.88±0.02μs
==================================================================== =============
[ 50.00%] ·· Building for conda-py3.6-six
[ 50.00%] ·· Benchmarking conda-py3.6-six
[ 75.00%] ··· Running bench_ufunc.ArgParsing.time_add_arg_parsing ok
[ 75.00%] ····
============================================================================ =============
arg_kwarg
---------------------------------------------------------------------------- -------------
((array(1.), array(2.)), {}) 1.11±0.03μs
((array(1.), array(2.), array(3.)), {}) 1.23±0.02μs
((array(1.), array(2.)), {'out': array(3.)}) 1.24±0.03μs
((array(1.), array(2.)), {'out': (array(3.),)}) 1.05±0.01μs
((array(1.), array(2.)), {'subok': True, 'where': True, 'out': array(3.)}) 1.26±0.01μs
((array(1.), array(2.)), {'subok': True}) 1.37±0.01μs
((array(1.), array(2.)), {'subok': True, 'where': True}) 1.32±0.02μs
((array(1.), array(2.), array(3.)), {'subok': True, 'where': True}) 1.44±0.01μs
============================================================================ =============
[100.00%] ··· Running ...ArgParsingReduce.time_add_reduce_arg_parsing ok
[100.00%] ····
==================================================================== =============
arg_kwarg
-------------------------------------------------------------------- -------------
((array([0., 1.]),), {}) 1.85±0.03μs
((array([0., 1.]), 0), {}) 2.00±0.01μs
((array([0., 1.]),), {'axis': 0}) 2.26±0.01μs
((array([0., 1.]), 0, None), {}) 1.98±0.02μs
((array([0., 1.]),), {'dtype': None, 'axis': 0}) 2.32±0.06μs
((array([0., 1.]), 0, None, array(0.)), {}) 2.06±0.02μs
((array([0., 1.]),), {'dtype': None, 'out': array(0.), 'axis': 0}) 2.50±0.02μs
((array([0., 1.]),), {'out': array(0.)}) 2.27±0.01μs
==================================================================== =============
|
Thanks Marten. |
|
@pv: Will this actually be able to compare benchmarks? How is one benchmark matched against another? Tuples of ndarrays are unlikely to compare equal... |
|
Each parameter set is treated as a separate benchmark, if that is what you are asking. |
As requested in #11351 (comment)