Codecov Report

Merging #3706 into master will decrease coverage by 21.31%.
The diff coverage is 57.14%.

@@             Coverage Diff             @@
##           master    #3706       +/-   ##
===========================================
- Coverage   87.79%   66.48%   -21.32%     
===========================================
  Files         688      323      -365     
  Lines       84280    48123    -36157     
  Branches     1081        0     -1081     
===========================================
- Hits        73994    31993    -42001     
- Misses      10175    16130     +5955     
+ Partials      111        0      -111

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e61bcf...eb5c8ba. Read the comment docs.

I'm surprised we're importing Pandas inconditionally. We probably shouldn't do that, as Pandas is quite slow to import:

$ time python -c "import numpy"

real	0m0,097s
user	0m0,077s
sys	0m0,020s

$ time python -c "import pandas"

real	0m0,300s
user	0m0,272s
sys	0m0,028s

Here is a comparison of PyArrow import time with and without Pandas:

$ time python -c "import pyarrow"

real	0m0,360s
user	0m0,305s
sys	0m0,037s

$ time python -c "import sys; sys.modules['pandas'] = None; import pyarrow"

real	0m0,144s
user	0m0,124s
sys	0m0,020s

=> more than twice faster without.

@pitrou Any ideas on how to avoid these local imports and also have the benefit of only loading pandas when needed?

Imports are reasonably cheap once the module is already loaded, but it's probably better to avoid doing them in a tight loop. So hoisting the import outside of critical loops should be sufficient.

>>> %timeit import pandas                                                                                                                                         
102 ns ± 0.497 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> %timeit import pyarrow.pandas_compat as pdcompat                                                                                                              
253 ns ± 7.27 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

This is in pure Python, though. Cython does not seem to implement the same optimizations as CPython does:

>>> %timeit lib._noop_bench()                                                                                                                                     
66.3 ns ± 0.379 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> %timeit lib._import_bench()                                                                                                                                   
928 ns ± 19.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

(edit: issue opened for Cython at cython/cython#2854)

@pitrou Are you happy with the change here and we will deal with the Pandas import issue separately or should this patch be adapted before merging?

Yes, I've created ARROW-4637 for the pandas import.

@xhochy @pitrou we could create a singleton object that exposes the pandas APIs that we need. The imports will be performed on creation of the singleton