Support empty syncs with no batches provided as payload #591
Conversation
| })? | ||
| } else { | ||
| None | ||
| SyncSchema::empty() |
There was a problem hiding this comment.
Can this actually legitimately happen now or should we always expect that there is a schema?
There was a problem hiding this comment.
It seems that providing an empty payload (vec![]) is easier than providing a non-empty payload with empty record batches (vec![RecordBatch::new_empty(schema)]), so I think yes.
Let's hear from @eatonphil though.
There was a problem hiding this comment.
It seems that providing an empty payload (vec![]) is easier than providing a non-empty payload with empty record batches (vec![RecordBatch::new_empty(schema)]), so I think yes.
Well, let me try. I have such a hard time with the Arrow APIs I was scared to try. But if you think that's better I'll give it a shot.
There was a problem hiding this comment.
It may still better of to have special casing of the empty payload, than to rely on the rest of the mechanism to adequately process those messages (so this change is for the better anyway).
There was a problem hiding this comment.
Alright, let's go with it then.
This is to signal a transaction end.
Also rename batch compaction to squashing throughout.