BinaryArrow Byte Limitation #1673
Description
I'm having some trouble converting a DataFrame into a table:
import numpy as np
import pandas as pd
import pyarrow as pa
x = list('1' * 2**31)
y = pd.DataFrame({'x': x})
t = pa.Table.from_pandas(y)
# ArrowInvalid: BinaryArrow cannot contain more than 2147483646 bytes, have 2147483647
This is for pyarrow 0.8.0. Should it be creating chunked arrays automatically for me? Ultimately, my goal is to write to parquet.