I don't believe this can be easily triggered by an end user but in case meta is improperly constructed internally (e.g. in dask-expr during optimization) and passed to the extension this can cause severe data loss.

The relevant section is this

In the example I am currently debugging, the data here looks like

while meta includes additional columns

convert_shards then raises a KeyError here

which causes _get_output_partition to return an empty dataframe. I haven't spent time figuring out why this exception is handled the way it is here but if this logic is required, the bare minimum we have to do is to verify columns of meta and payload data before transmission.

cc @hendrikmakait