https://issues.apache.org/jira/browse/ARROW-13740

@bkietz @lidavidm could either of you check out this debug check failure: https://github.com/apache/arrow/pull/10992/checks?check_run_id=3437174950#step:9:12942 ?

@jonkeane can you help me figure out what's up with https://github.com/apache/arrow/pull/10992/checks?check_run_id=3437175048#step:8:15613?

@bkietz @lidavidm could either of you check out this debug check failure: https://github.com/apache/arrow/pull/10992/checks?check_run_id=3437174950#step:9:12942 ?

It looks like two targets are being passed in but only one aggregation. some_grouping probably shouldn't go into the targets?

(gdb) p aggs.size()
$2 = 1
(gdb) p aggregate_options.targets.size()
$3 = 2
(gdb) p aggregate_options.targets[0].ToString()
$5 = {static npos = 18446744073709551615, 
  _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
    _M_p = 0x55555a939db0 "FieldRef.Name(total)"}, _M_string_length = 20, {
    _M_local_buf = "\036\000\000\000\000\000\000\000\277j\366\377\377\177\000", _M_allocated_capacity = 30}}
(gdb) p aggregate_options.targets[1].ToString()
$6 = {static npos = 18446744073709551615, 
  _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
    _M_p = 0x55555a939de0 "FieldRef.Name(some_grouping)"}, _M_string_length = 28, {
    _M_local_buf = "\036\000\000\000\000\000\000\000\277j\366\377\377\177\000", _M_allocated_capacity = 30}}

@bkietz @lidavidm could either of you check out this debug check failure: https://github.com/apache/arrow/pull/10992/checks?check_run_id=3437174950#step:9:12942 ?

It looks like two targets are being passed in but only one aggregation. some_grouping probably shouldn't go into the targets?

Thanks, it looks like I've fixed this (unintentionally)

I've fixed the second of the two metadata failures (that had hard-coded reordering, which I changed to arrange.)

The one about the missing warning is trickier / I haven't fully solved it yet. What's going on there is that the dataset has metadata associated with it, but when we run https://github.com/apache/arrow/pull/10992/files#diff-fee018727db3ca05257dd7755794b240a3974e7ee4d6c1803a2d9efeb9d692a9R39

collect.Dataset <- function(x, ...) dplyr::collect(as_adq(x), ...)

as_adq(x) results in an object that does not have an element named metadata anymore (x$.data$metadata) still exists, of course.

The one about the missing warning is trickier / I haven't fully solved it yet.

I think what's happening is that the Dataset's schema metadata isn't propagated through the ExecPlan, while it was through the Scanner. I changed how collect() works in this PR to use ExecPlan so that it could also handle aggregations and sorting.

I'll skip the test for now and make a JIRA. It's not a problem in this test (because actually we want the metadata to be dropped) but it would be a problem for other contexts where we have r metadata.

Re: the metadata issue, the Jira is https://issues.apache.org/jira/browse/ARROW-13852 FTR here.

I do worry a little bit that this could break folk's workflows in surprising and frustrating ways. I'm going to bump the priority on that Jira to blocker since we definitely don't want to release without it done.

Looks like 32-bit rtools35 just doesn't work with the async scanning stuff (if I recall past tests that we've had to skip).