Currently, when reading a subfield of a nested column of a Parquet file using the Dataset API, we read the full parent column instead of only the requested field. This should be optimized to only read the field itself.

This was left as a TODO in ARROW-14658 (#11704) which added the initial support for nested field refs in dataset scanning (

  if (field) {
    // TODO(ARROW-1888): support fine-grained column projection. We should be
    // able to materialize only the child fields requested, and not the entire
    // top-level field.
    // Right now, if enabled, projection/filtering will fail when they cast the
    // physical schema to the dataset schema.
    AddColumnIndices(*toplevel, columns_selection);

Some relevant comments at #11704 (comment). ARROW-1888 was mentioned as a blocker back then, but this is resolved in the meantime.

Reporter: Joris Van den Bossche / @jorisvandenbossche

_{Note: This issue was originally created as ARROW-17959. Please see the migration documentation for further details.}