Describe the bug
Using delta-rs to write to gcp configured s3 storage yields a 411 error.
Relevant ticket in delta-rs: delta-io/delta-rs#2394
To Reproduce
delta-io/delta-rs#2394 (comment)
Expected behavior
Should be able to use gcp configured s3 storage.
Additional context
I did some digging and it appears this may have been fixed at one point by apache/arrow-rs#4518
but was then regressed by apache/arrow-rs@8731760#diff-0feb099a0931110c214fa9b9c8c5528512f38ebf22448772a815baee5973b8c4L354-L357
I also noticed that the aws client doesn't currently appear to pass content length of zero:
https://github.com/apache/arrow-rs/blob/main/object_store/src/aws/client.rs#L615
Yet the gcp client does: https://github.com/apache/arrow-rs/blob/main/object_store/src/gcp/client.rs#L441
Describe the bug
Using
delta-rsto write to gcp configured s3 storage yields a 411 error.Relevant ticket in delta-rs: delta-io/delta-rs#2394
To Reproduce
delta-io/delta-rs#2394 (comment)
Expected behavior
Should be able to use gcp configured s3 storage.
Additional context
I did some digging and it appears this may have been fixed at one point by apache/arrow-rs#4518
but was then regressed by apache/arrow-rs@8731760#diff-0feb099a0931110c214fa9b9c8c5528512f38ebf22448772a815baee5973b8c4L354-L357
I also noticed that the aws client doesn't currently appear to pass content length of zero:
https://github.com/apache/arrow-rs/blob/main/object_store/src/aws/client.rs#L615
Yet the gcp client does: https://github.com/apache/arrow-rs/blob/main/object_store/src/gcp/client.rs#L441