Skip to content

ARROW-2074: [Python] Infer lists of dicts as struct arrays #1935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pitrou wants to merge 2 commits into from
Closed

ARROW-2074: [Python] Infer lists of dicts as struct arrays #1935

pitrou wants to merge 2 commits into from

Conversation

@pitrou
Copy link
Member

@pitrou pitrou commented Apr 23, 2018 •
edited
Loading

Also refactor the type inference visitor and remove the superfluous separate SeqVisitor; improve inference visitor performance by 30%; and add a struct type inference benchmark.

@pitrou
ARROW-2074: [Python] Infer lists of dicts as struct arrays
3baa2ea 
Also refactor the type inference visitor, improve visitor performance by ~30%,
and a benchmark for struct type inference.
@pitrou pitrou force-pushed the ARROW-2074-infer-dict-lists branch from bfc1f3b to 3baa2ea Compare April 23, 2018 15:32
@pitrou
Copy link
Member Author

pitrou commented Apr 23, 2018

Benchmark numbers here:

  • before:
[100.00%] ··· Running convert_builtins.InferPyListToArray.time_infer                                                                                     ok
[100.00%] ···· 
               ============ =============
                   type                  
               ------------ -------------
                  int64       11.0±0.1ms 
                 float64     10.3±0.07ms 
                   bool      9.37±0.04ms 
                 decimal      297±0.9ms  
                  binary      14.9±0.2ms 
                  ascii       17.3±0.3ms 
                 unicode      29.7±0.8ms 
                int64 list    96.8±0.6ms 
               ============ =============
  • after:
[100.00%] ··· Running convert_builtins.InferPyListToArray.time_infer                                                                                     ok
[100.00%] ···· 
               ============ =============
                   type                  
               ------------ -------------
                  int64       7.41±0.2ms 
                 float64     6.68±0.04ms 
                   bool      5.75±0.01ms 
                 decimal      292±0.8ms  
                  binary      11.4±0.2ms 
                  ascii       14.1±0.3ms 
                 unicode      26.3±0.7ms 
                int64 list    74.8±0.6ms 
                  struct       70.7±4ms  
               ============ =============

@pitrou pitrou mentioned this pull request Apr 24, 2018
Closed
xhochy
xhochy approved these changes Apr 25, 2018
View reviewed changes
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

@xhochy xhochy closed this in 3d7a5a6 Apr 25, 2018
@pitrou pitrou deleted the ARROW-2074-infer-dict-lists branch April 25, 2018 23:02
pitrou added a commit that referenced this pull request May 1, 2018
@pitrou
ARROW-2499: [C++] Factor out Python iteration routines
2bc110d 
Speeds up list to Arrow conversions by up to 15%. Also fixes a bug where creating a list array would not check that all input items are sequences.

Based on PR #1935.

Author: Antoine Pitrou <antoine@python.org>

Closes #1940 from pitrou/ARROW-2499-python-iteration-refactor and squashes the following commits:

ac31c6c <Antoine Pitrou> Fix Ndarray1DIndexer::is_strided (unused)
91c5af1 <Antoine Pitrou> Add TODO for performance issue
00cab9a <Antoine Pitrou> ARROW-2499:  Refactor Python iteration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xhochy xhochy xhochy approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou @xhochy