Calling pa.nulls on a nullable fixed size list of non-nullable floats (i.e. Vec<Option<[f32; N]>>) actually generates a nullable fixed size list of nullable floats (i.e. Vec<Option<[Option<f32>; N]>>), which breaks* upon deserialization since it doesn't match the expected type.
[*] in a way that is extremely hard to track down as the type will show up as semantically identical... ⚠️
#2307 introduced a workaround directly within the python SDK so that we generate the right thing, but that won't scale.
I would rather not blindly ignore passed in bitmaps if the expected datatype is non-nullable... but on the other hand it might be okay to silently ignored the passed-in bitmap if and only if the datatype specified by the sender itself is non-nullable.
Calling
pa.nullson a nullable fixed size list of non-nullable floats (i.e.Vec<Option<[f32; N]>>) actually generates a nullable fixed size list of nullable floats (i.e.Vec<Option<[Option<f32>; N]>>), which breaks* upon deserialization since it doesn't match the expected type.[*] in a way that is extremely hard to track down as the type will show up as semantically identical...⚠️
#2307 introduced a workaround directly within the python SDK so that we generate the right thing, but that won't scale.
I would rather not blindly ignore passed in bitmaps if the expected datatype is non-nullable... but on the other hand it might be okay to silently ignored the passed-in bitmap if and only if the datatype specified by the sender itself is non-nullable.