Describe the bug, including details regarding any error messages, version, and platform.
A gRPC issue has been identified where transmitting buffers of zero length leads to a persistent hang on MacOS/iOS platforms.
Such zero-length buffers may arise, for example, in the context of using the Array structure alongside a validity bitmap, as outlined in the Arrow Spec.
Essentially, when every element within an Array is valid (i.e., not null), it's possible to represent this state with a null validity bitmap, indicating that all elements are valid. This scenario is realized through the use of a null buffer, as demonstrated here:
|
// In V4, null types have no validity bitmap |
|
// In V5 and later, null and union types have no validity bitmap |
|
if (internal::HasValidityBitmap(arr.type_id(), options_.metadata_version)) { |
|
if (arr.null_count() > 0) { |
|
std::shared_ptr<Buffer> bitmap; |
|
RETURN_NOT_OK(GetTruncatedBitmap(arr.offset(), arr.length(), arr.null_bitmap(), |
|
options_.memory_pool, &bitmap)); |
|
out_->body_buffers.emplace_back(std::move(bitmap)); |
|
} else { |
|
// Push a dummy zero-length buffer, not to be copied |
|
out_->body_buffers.emplace_back(kNullBuffer); |
|
} |
|
} |
|
return VisitType(arr); |
|
} |
The relevant sections of code from both transport mechanisms are provided below for reference:
|
// Enqueue body buffers for writing, without copying |
|
for (const auto& buffer : ipc_msg.body_buffers) { |
|
// Buffer may be null when the row length is zero, or when all |
|
// entries are invalid. |
|
if (!buffer) continue; |
|
for (const auto& buffer : payload.ipc_message.body_buffers) { |
|
if (!buffer || buffer->size() == 0) continue; |
Upon examining the UCX transport's approach, it's evident that a precaution is taken to avoid sending zero-length buffers. This strategy appears prudent, as it eliminates the need to forward non-transmittable buffers to the transport layer, potentially offering a solution to the issue observed with gRPC.
Component(s)
C++, FlightRPC
Describe the bug, including details regarding any error messages, version, and platform.
A gRPC issue has been identified where transmitting buffers of zero length leads to a persistent hang on MacOS/iOS platforms.
Such zero-length buffers may arise, for example, in the context of using the Array structure alongside a validity bitmap, as outlined in the Arrow Spec.
Essentially, when every element within an Array is valid (i.e., not null), it's possible to represent this state with a null validity bitmap, indicating that all elements are valid. This scenario is realized through the use of a null buffer, as demonstrated here:
arrow/cpp/src/arrow/ipc/writer.cc
Lines 165 to 179 in 187197c
The relevant sections of code from both transport mechanisms are provided below for reference:
arrow/cpp/src/arrow/flight/transport/grpc/serialization_internal.cc
Lines 283 to 287 in 187197c
arrow/cpp/src/arrow/flight/transport/ucx/ucx_internal.cc
Lines 590 to 591 in 187197c
Upon examining the UCX transport's approach, it's evident that a precaution is taken to avoid sending zero-length buffers. This strategy appears prudent, as it eliminates the need to forward non-transmittable buffers to the transport layer, potentially offering a solution to the issue observed with gRPC.
Component(s)
C++, FlightRPC