ARROW-1572: [C++] Implement "value counts" kernels for tabulating value frequencies #3579
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thanks for picking this up from where I left off, I'll close my original PR. |
Contributor
Author
|
@andrioni Thanks for the contribution |
62d23fd to
9a5c9ca
Compare
fsaintjacques
approved these changes
Feb 8, 2019
Contributor
fsaintjacques
left a comment
There was a problem hiding this comment.
Some notes:
- May I propose to rename this to
Distribution? - We should also count Nulls.
- The current output format is 2 arrays. I think we should formalize this and return a single array of
List<Struct<Option<value>, count>>explicitly which should allow returning the count for the null value.
This can be done in another ticket/PR.
Contributor
There was a problem hiding this comment.
For completeness, you should set the datum type if we ever refactor.
Contributor
Author
There was a problem hiding this comment.
Could you clarify? I might be missing it but I don't see an API to set the type explicitly (instead it looks like it gets derived when value that is assigned in the underlying variant)
Member
|
Rebased |
Contributor
Author
|
@fsaintjacques thanks for the review. My thoughts:
|
30dc7d5 to
8779edd
Compare
emkornfield
commented
Feb 12, 2019
cpp/src/arrow/compute/kernels/hash.h
Outdated
Contributor
Author
There was a problem hiding this comment.
this could potentially return a StructArray directly.
8779edd to
8c77e19
Compare
193d110 to
a6b9364
Compare
a24af0b to
123c6c0
Compare
Contributor
Author
|
@fsaintjacques I think I fixed the windows build (will check in the morning). I changed the output to be a Struct as suggested. I will file follow-up JIRAs for the rest. |
cpp/src/arrow/util/hashing.h
Outdated
Contributor
There was a problem hiding this comment.
I'd put a DCHECK_GE(start, 0)` just to make sure it's positive.
Contributor
Author
There was a problem hiding this comment.
Also put a ASSERT_LE in and changed to std::copy which i think fixes another bug.
0176819 to
ae08e40
Compare
Contributor
Author
|
Clean build 👍 |
Contributor
|
I want to give a last review, but I don't think anything is blocking. |
fsaintjacques
approved these changes
Mar 1, 2019
Contributor
There was a problem hiding this comment.
Make Values and Count a public constexpr such that user can programatically retrieve the field by name at compile time, maybe expose the index that way too.
Contributor
Author
There was a problem hiding this comment.
Good suggestion. Done.
Contributor
|
Ready to merge! |
Contributor
|
I think you need to rebase for the failing appveyor failure, but I'd say this is safe to merge. LGTM +1. |
7825282 to
2845381
Compare
Contributor
Author
|
One thing that I'll do which I'm not sure will help too much is separate out the memcpy fix into it's own PR so we can isolate performance problems to the kernel |
Contributor
Author
|
I think there might also be a way forward with template magic to eliminate this altogether. I'll get an update out soon |
cdd6638 to
72095eb
Compare
Contributor
Author
|
With this change it looks like they are fairly close on my box (rerunning causes changes in different directions in different metrics)
|
Member
|
Looking at this again and then merging, thanks for your patience |
Member
|
I ran on a mobile Xeon 2.8 ghz with the cores pinned at 2.8ghz / no turbo boost and things looking OK here before after merging. thanks @emkornfield! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Picked up from: #1970
I mostly reused the unit tests as is and modified the rest based on feedback on that PR and adapting to new code. Some other changes I mae:
If this approach looks OK for now. I'm also going to open up a JIRA to try refactor the unittest for Hash kernels (and maybe the headers) since I think there might be a clearer more granular way of laying these out.
other things to discuss: