Skip to content

Support deserializing illegally nullable data #2871

@teh-cmc

Description

@teh-cmc

Calling pa.nulls on a nullable fixed size list of non-nullable floats (i.e. Vec<Option<[f32; N]>>) actually generates a nullable fixed size list of nullable floats (i.e. Vec<Option<[Option<f32>; N]>>), which breaks* upon deserialization since it doesn't match the expected type.

[*] in a way that is extremely hard to track down as the type will show up as semantically identical... ⚠️

#2307 introduced a workaround directly within the python SDK so that we generate the right thing, but that won't scale.

I would rather not blindly ignore passed in bitmaps if the expected datatype is non-nullable... but on the other hand it might be okay to silently ignored the passed-in bitmap if and only if the datatype specified by the sender itself is non-nullable.

Metadata

Metadata

Assignees

Labels

🏹 arrowApache Arrow😤 annoyingSomething in the UI / SDK is annoying to use🪳 bugSomething isn't working

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions