Summary

Two fixes to enable reading Parquet files from S3.

1. read_parquet doesn't work with S3 URLs

In [2]: pd.read_parquet('s3://datasets-documentation/amazon_reviews/amazon_reviews_2015.snappy.parquet')
Out[2]: E [chDB] Query failed: Code: 400. DB::ErrnoException: Cannot stat file /Users/markhneedham/projects/videos/20241029-chdbPandas/s3:/datasets-documentation/amazon_reviews/amazon_reviews_2015.snappy.parquet: , errno: 2, strerror: No such file or directory: The table structure cannot be extracted from a Parquet format file. You can specify the structure manually: (in file/uri /Users/markhneedham/projects/videos/20241029-chdbPandas/s3:/datasets-documentation/amazon_reviews/amazon_reviews_2015.snappy.parquet). (CANNOT_STAT)
...
SQL: DESCRIBE file('s3://datasets-documentation/amazon_reviews/amazon_reviews_2015.snappy.parquet', 'Parquet')

s3:// paths were being routed to from_file(), which treats them as local paths. Fixed by routing s3:// paths in read_parquet() to DataStore.from_s3() instead.

2. DataStore.from_s3() crashes without an explicit format

In [3]: pd.DataStore.from_s3('s3://datasets-documentation/amazon_reviews/amazon_reviews_2015.snappy.parquet', nosign=True)
Out[3]: DataStore(execution failed: 'NoneType' object has no attribute 'lower')

Root cause: from_s3() stores {"format": None} in table function params when no format is specified. dict.get("format", "") returns None (not "") when the key exists with value None — the default only applies when the key is absent entirely. None.lower() then crashes in preserves_row_order().

Fixed by using (self.params.get("format") or "").lower() in both FileTableFunction and S3TableFunction.

Changes

• datastore/pandas_api.py: Route s3:// paths in read_parquet() to DataStore.from_s3()
• datastore/table_functions.py: Fix None.lower() crash in FileTableFunction.preserves_row_order() and S3TableFunction.preserves_row_order()

Test plan

• read_parquet("s3://...") routes to S3 table function instead of crashing
• DataStore.from_s3("s3://...", nosign=True) no longer raises 'NoneType' object has no attribute 'lower'
• DataStore.from_file("data.parquet") with no explicit format still works

🤖 Generated with Claude Code