I came through a native memory leak (#884) caused by the misuse of the Java Implementation of the Arrow C data interface. I'm wondering why Datafusion Comet is not following the memory management semantics stated in the Arrow C Data Interface Specification? The specification says that:

Member allocation

It is intended for the base structure to be stack- or heap-allocated by the consumer. In this case, the producer API should take a pointer to the consumer-allocated structure.

The base structures (ArrowSchema and ArrowVector) should be allocated by the consumer of Arrow vectors, the consumer passes 2 memory locations to the producer, and producer fills the base structures with pointers to the buffers allocated by the producer, as well as a release callback function for freeing up the resources allocated by the producer.

Datafusion Comet always allocates base structures in the producer and returns the base structures to the consumer. The producer may have a hard time figuring out when to deallocate the base structures: it has to happen after the consumer moves the Arrow vectors and takes ownership of them:

• On the native side, the allocated base structures were kept in the ffi_arrays field of the execution context, and will be released when the next batch is produced or when the execution context is released by Native.releasePlan.
• On the JVM side, the lifetime of allocated base structures was not taken care of, thus causing native memory leaks.

If we follow the memory management semantics stated by the specification, we can always free the base structures immediately after calling the producer and importing the vector, thus having a simpler model of base structure lifetime, and reducing the cost of adapting the mindset from one memory management semantics to another for contributors familiar with the Arrow C Data Interface.