fix: Ignore nullability of list elements when consuming Substrait #10874
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of apache#10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ```
just for consistency, to reduce the places where "item" is written out
Blizzara
commented
Jun 11, 2024
| let offsets = OffsetBuffer::from_lengths([arr.len()]); | ||
| ListArray::new( | ||
| Arc::new(Field::new("item", arr.data_type().to_owned(), true)), | ||
| Arc::new(Field::new_list_field(arr.data_type().to_owned(), true)), |
Contributor
Author
There was a problem hiding this comment.
this is no-op change, just seems nicer to use the new_list_field given it exists (it sets the name as "item" anyways)
Contributor
|
The CI failure appears to be unrelated to this PR, so I restarted the tests |
alamb
approved these changes
Jun 11, 2024
| let offsets = OffsetBuffer::from_lengths([arr.len()]); | ||
| LargeListArray::new( | ||
| Arc::new(Field::new("item", arr.data_type().to_owned(), true)), | ||
| Arc::new(Field::new_list_field(arr.data_type().to_owned(), true)), |
| let field = Arc::new(Field::new_list_field( | ||
| from_substrait_type(inner_type, dfs_names, name_idx)?, | ||
| is_substrait_type_nullable(inner_type)?, | ||
| // We ignore Substrait's nullability here to match to_substrait_literal |
| Field::new_list_field(DataType::Int32, nullable).into(), | ||
| ))?; | ||
| } | ||
| round_trip_type(DataType::List( |
Contributor
There was a problem hiding this comment.
Should we also add a test here showing that the List becomes nullable after roundtrip? That might additionally document that this is an intended behavior rather than a bug.
I do see you have added a comment which I think is probably enought
Contributor
Author
There was a problem hiding this comment.
I don't think we need to test the null coercion specifically, since that's not necessarily the ultimate desired behavior, just what makes sense at this time. But I added a test case to confirm that we can now read plans with non-nullable lists: a25abd8
Contributor
|
Thanks again @Blizzara |
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…ache#10874) * Ignore nullability of list elements when consuming Substrait DataFusion (= Arrow) is quite strict about nullability, specifically, when using e.g. LogicalPlan::Values, the given schema must match the given literals exactly - including nullability. This is non-trivial to do when converting schema and literals separately. The existing implementation for from_substrait_literal already creates lists that are always nullable (see ScalarValue::new_list => array_into_list_array). This reverts part of apache#10640 to align from_substrait_type with that behavior. This is the error I was hitting: ``` ArrowError(InvalidArgumentError("column types must match schema types, expected List(Field { name: \"item\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: \"item\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0"), None) ``` * use `Field::new_list_field` in `array_into_(large_)list_array` just for consistency, to reduce the places where "item" is written out * add a test for non-nullable lists
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DataFusion (or really Arrow) is quite strict about nullability, specifically, when using e.g.
LogicalPlan::Values, the given schema must match the given literals exactly - including nullability.This is non-trivial to do when converting schema and literals separately like we do.
The existing implementation for from_substrait_literal already creates lists that are always nullable
(see ScalarValue::new_list => array_into_list_array). This reverts part of #10640 to align from_substrait_type with that behavior.
This is the error I was hitting:
Which issue does this PR close?
Closes #.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Tested through existing unit tests + manually the failing case I had.
Are there any user-facing changes?
No