This issue tracks progress for adding the apache arrow parquet library and integrating it into changefeeds.

https://github.com/apache/arrow/

This decision was based on the investigation done in this doc. In summary

• The existing implementation of changefeed initial scans with parquet is 15x slower than JSON
• CPU flamegraphs show that 53% of time is spent in the current library and an additional ~20% is doing GC
• An initial investigation of the old and new libraries shows that the new one would be more efficient with handling memory and writing files. We plan on integrating the new library with the smallest amount of code changes to enable the cdc TPCC roachtest, which we can use to benchmark the new library and confirm our observations.

I'll be working on the following items roughly in order

• Install dependency (cdc: add apache arrow parquet library and writer #99288)
• Create writer library ( cdc: add apache arrow parquet library and writer #99288)
• Add support for tpcc types ( cdc: add apache arrow parquet library and writer #99288)
• Benchmark new parquet library with tpcc roachtest (results )
• Implement reader library and tests for tpcc types (cdc: add apache arrow parquet library and writer #99288)
• Add support for all types
  • util/parquet: add support for arrays #101860
  • util/parquet: add support for all types #101946)
• Add compression options to writer util/parquet: add compression options #102978
• Add memory accounting This is now tracked by export: parquet causes node to OOM #103317
  • See https://github.com/cockroachlabs/support/issues/2247
• Using library for EXPORT INTO tracked by export: replace parquet library for EXPORT  #103318
• Integrate writer with cdc
  • util/parquet: add crdb metadata to parquet files #103131
  • changefeedccl: use new parquet library #103293
  • Revert "do not merge: always force parquet format in cdc tests" #103928
• Add support for tuples, which is needed for diff and cdc_prev.
  • util/parquet: refactor random testing types #103617
  • util/parquet: support tuples #103589
• Add previously unsupported options to parquet changefeedccl: remove limitations for parquet format #103129
  • changefeedccl: support the resolved option with format=parquet #104283
  • changefeedccl: support the updated and mvcc with parquet format #104407
  • util/parquet: copy bytes from fmtCtx when writing byte arrays #104626
  • changefeedccl: add full support for the parquet format #104528
• Ensure flushing is efficient and add metrics
  • changefeedccl: improve parquet byte estimates and flushing scheme #111138

Jira issue: CRDB-25669

Epic CRDB-27372