@alamb Apologies, I should have been more clear. The Parquet file mentioned was in #2439. Attaching here as well:
part-00000-f6337bce-7fcd-4021-9f9d-040413ea83f8-c000.snappy.parquet.zip

The above mentioned error was when I ran the SQL from ballista and I've checked that ballista on the master branch is currently using datafusion version 18.0.0.

Hence, I just wrote two little tests, using the datafusion and the ballista context respectively.

SQL from the datafusion context works, whilst the one that uses the ballista context fails. Test code:

#[tokio::test]
async fn test_datafusion_sql() {
    let ctx = SessionContext::new();
    let filename = "part-00000-f6337bce-7fcd-4021-9f9d-040413ea83f8-c000.snappy.parquet";
    ctx.register_parquet("t", filename, ParquetReadOptions::default()).await.unwrap();
    let df = ctx.sql("select t.text['status'] from t").await.unwrap();
    df.show().await.unwrap();
}

Output:

+----------------+
| t.text[status] |
+----------------+
|                |
| generated      |
| generated      |
| generated      |
| generated      |
| generated      |
| generated      |
| generated      |
| generated      |
| generated      |
+----------------+

#[tokio::test]
async fn test_ballista_sql() {
    let config = BallistaConfig::builder().build().unwrap();
    let ctx = BallistaContext::standalone(&config, 10).await.unwrap();
    let filename = "part-00000-f6337bce-7fcd-4021-9f9d-040413ea83f8-c000.snappy.parquet";
    ctx.register_parquet("t", filename, ParquetReadOptions::default()).await.unwrap();
    let df = ctx.sql("select t.text['status'] from t").await.unwrap();
    df.show().await.unwrap();
}

Output:

thread 'query::test::test_ballista_sql' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(Execution("Job QeRwZCh failed: Error planning job QeRwZCh: DataFusionError(Internal(\"physical_plan::to_proto() unsupported expression GetIndexedFieldExpr { arg: Column { name: \\\"text\\\", index: 0 }, key: Utf8(\\\"status\\\") }\"))")))', src/query.rs:44:25

Please note that the error is coming from datafusion/proto/src/physical_plan/to_proto.rs and not the logical_plan.

Here are the relevant parts of my Cargo.toml.

ballista = { git = "https://github.com/apache/arrow-ballista", features = ["s3"] }
ballista-cli = { git = "https://github.com/apache/arrow-ballista", features = ["s3"] }
ballista-core = { git = "https://github.com/apache/arrow-ballista", features = ["s3"] }
datafusion = "18.0.0"

futures = "0.3"
object_store = "0.5"
tokio = { version = "1", features = ["full"] }

Feature request for support for struct and arry data types #3617

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions