Simplify the codebase by avoiding metaprogramming #696
Description
Following on from #683 (comment) concerning the speed of creating a context object and see the past discussions in gh-178.
The mpmath codebase uses a number of metaprogramming features that increase complexity for what I suspect is relatively little benefit (mostly DRY and micro-optimisation). These make the codebase hard to follow and also defeat static code analysis which is increasingly integrated in modern Python editors. I propose to remove these and move towards ordinary Python code with plain functions, methods and attributes.
Specifically I suggest:
- Eliminate dynamic class generation. Make e.g. mpf a simple type that just has
_mpf_andctxattributes. - Write methods in ordinary Python code rather than generating them dynamically either with exec or by attaching them from decorators.
- Use more traditional methods for reducing boilerplate like reusable functions.
One example of this is dynamically created classes like:
mpmath/mpmath/ctx_mp_python.py
Lines 584 to 596 in 29b47ce
There are also functions and methods created from code strings at runtime:
mpmath/mpmath/ctx_mp_python.py
Lines 280 to 293 in 29b47ce
We also have methods that are dynamically added to classes:
mpmath/mpmath/functions/functions.py
Lines 16 to 20 in 29b47ce
Some of these happen at import time and others happen when a context object is created and add measurable overhead to the cost of creating a context:
In [9]: from mpmath import MPContext, mp
In [10]: %timeit MPContext()
538 µs ± 16.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
In [11]: %timeit mp.clone()
561 µs ± 27.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
In [12]: %timeit mp.cos(1)
10.9 µs ± 253 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [18]: %prun -s cumulative [mp.clone() for _ in range(1000)]
863004 function calls in 1.736 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.736 1.736 {built-in method builtins.exec}
1 0.000 0.000 1.736 1.736 <string>:1(<module>)
1 0.002 0.002 1.736 1.736 <string>:1(<listcomp>)
1000 0.005 0.000 1.734 0.002 ctx_mp.py:297(clone)
1000 0.022 0.000 1.718 0.002 ctx_mp.py:63(__init__)
1000 0.007 0.000 0.831 0.001 ctx_base.py:42(__init__)
1000 0.364 0.000 0.783 0.001 functions.py:18(__init__)
1000 0.355 0.000 0.775 0.001 ctx_mp.py:96(init_builtins)
213000 0.270 0.000 0.378 0.000 ctx_mp_python.py:1014(_wrap_specfun)
43000 0.356 0.000 0.372 0.000 ctx_mp_python.py:979(_wrap_libmp_function)
220000 0.080 0.000 0.080 0.000 {built-in method builtins.setattr}
1000 0.073 0.000 0.073 0.000 ctx_mp_python.py:585(__init__)
256000 0.048 0.000 0.048 0.000 {method 'get' of 'dict' objects}
17000 0.020 0.000 0.038 0.000 rational.py:31(__new__)
14000 0.017 0.000 0.037 0.000 ctx_mp_python.py:336(__new__)
1000 0.023 0.000 0.023 0.000 matrices.py:749(__init__)
26000 0.020 0.000 0.020 0.000 {built-in method builtins.getattr}
17000 0.016 0.000 0.018 0.000 rational.py:7(create_reduced)
1000 0.008 0.000 0.015 0.000 ctx_base.py:52(_init_aliases)
24000 0.011 0.000 0.011 0.000 {built-in method __new__ of type object at 0x7f2bc4270a40}
1000 0.005 0.000 0.010 0.000 ctx_mp_python.py:612(_set_prec)
1000 0.008 0.000 0.010 0.000 inverselaplace.py:668(__init__)
1000 0.004 0.000 0.006 0.000 quadrature.py:461(__init__)
5000 0.003 0.000 0.006 0.000 ctx_mp_python.py:597(make_mpf)
1000 0.003 0.000 0.004 0.000 libmpf.py:59(prec_to_dps)
1000 0.002 0.000 0.002 0.000 ctx_base.py:458(memoize)
1000 0.002 0.000 0.002 0.000 rszeta.py:54(__init__)
2000 0.002 0.000 0.002 0.000 quadrature.py:21(__init__)
2000 0.002 0.000 0.002 0.000 {built-in method builtins.max}
1000 0.001 0.000 0.001 0.000 ctx_mp_python.py:602(make_mpc)
1000 0.001 0.000 0.001 0.000 {built-in method builtins.round}
4000 0.001 0.000 0.001 0.000 inverselaplace.py:17(__init__)
1000 0.001 0.000 0.001 0.000 ctx_mp_python.py:620(<lambda>)
1000 0.001 0.000 0.001 0.000 ctx_mp_python.py:607(default)
1000 0.001 0.000 0.001 0.000 {method 'update' of 'dict' objects}
1000 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}Ideally the cost of creating a new context object would be small enough that it could be worthwhile to create one just for the purposes of a single function call. The context object only has a handful of genuine attributes so it should not be this expensive to create:
In [29]: class Context:
...: __slots__ = ('prec',)
...: def __init__(self, prec=53):
...: self.prec = prec
...:
In [30]: %timeit Context()
235 ns ± 25.1 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)I think that it would be better to rewrite these things using ordinary Python code with a primary emphasis on keeping the code simple and easy to understand and maintain. Future versions of CPython will also boost the performance of much Python code but mpmath will potentially not be able to benefit as much from these optimisations because they are generally targeted at simple straight-forward code.
With a simpler codebase it would also be easier to aim for bigger performance improvements by making it easier to swap different backends in and out and to leverage e.g. python_flint if available. The small performance benefits that are achieved by this metaprogramming could be made a lot greater just by having an optional C/Cython backend at least for basic types like mpf but it any such backend in a non dynamically typed language would not be able to replicate the metaprogramming. The python_flint module could house a faster version of mpf and MPContext if it was to be used as an optional backend for mpmath.