https://issues.apache.org/jira/browse/ARROW-10403

@nealrichardson what do you think about this approach? It introduces overhead to because it transposes dictionary indices but it gives us value_counts.

I'm not familiar with this C++ code so I'll let others comment (cc @pitrou @bkietz @michalursa). It looks like the issue is only with ChunkedArrays where the chunks have different dictionaries? My instinct is that, rather than unifying first and then determining unique values/counting/hashing, what if we could do the aggregation on each chunk first and then unify the results? That would be a smaller amount of data to manipulate.

My instinct is that, rather than unifying first and then determining unique values/counting/hashing, what if we could do the aggregation on each chunk first and then unify the results? That would be a smaller amount of data to manipulate.

Indeed unifying over all chunks first and then transposing individual chunk indices would be a better idea!

I'm still a bit unfamiliar with kernel mechanics but I'm thinking implementing a new kernel for chunked DictionaryArrays with different dictionaries will be the best way to go for this.

There are indeed two possible approaches:

• unify all chunks first, and then run the unique kernel over the transposed incides (as proposed by @rok)
• run the unique kernel over the original chunks, and then hash-aggregate the unique results of the different chunks (in effect SELECT sum(counts) GROUP BY values)

The second approach could be faster in the (unusual?) cases where only a small subset of dictionary values actually appear in the data. If most dictionary values are used, both cases should have similar performance, though.

Since we don't have a generic hash-aggregate yet, the first approach sounds good enough.
(also note that unique is in itself a special case of hash-aggregation)

cc @bkietz for opinions

Since we don't have a generic hash-aggregate yet, the first approach sounds good enough.
(also note that unique is in itself a special case of hash-aggregation)

Shall I then fix CI issues and we proceed with the first approach?
Or do we rather put effort into generic hash-aggregate?

You can fix CI issus IMHO.

This is ready for review - the java issue appears to be a flaky upload.

ping :)

Thanks @pitrou!
Should we make a jira for the generic hash aggregate?

Yes, please do!

https://issues.apache.org/jira/browse/ARROW-12301