-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Describe the bug
The FlightSQL schema definition for CommandGetDbSchemas and CommandGetTables responses does not fully correspond to the protocol due to missing nullability information
CommandGetDbSchemas
catalog_name is expected to be nullable:
arrow-rs/arrow-flight/src/sql/arrow.flight.protocol.sql.rs
Lines 130 to 137 in 7211502
| /// The returned Arrow schema will be: | |
| /// < | |
| /// catalog_name: utf8, | |
| /// db_schema_name: utf8 not null | |
| /// > | |
| /// The returned data should be ordered by catalog_name, then db_schema_name. | |
| #[derive(Clone, PartialEq, ::prost::Message)] | |
| pub struct CommandGetDbSchemas { |
Current definition
/// The schema for GetDbSchemas
static GET_DB_SCHEMAS_SCHEMA: Lazy<SchemaRef> = Lazy::new(|| {
Arc::new(Schema::new(vec![
Field::new("catalog_name", DataType::Utf8, false),
Field::new("db_schema_name", DataType::Utf8, false),
]))
});CommandGetTables
catalog_name and db_schema_name are expected to be nullable:
arrow-rs/arrow-flight/src/sql/arrow.flight.protocol.sql.rs
Lines 159 to 165 in 7211502
| /// The returned Arrow schema will be: | |
| /// < | |
| /// catalog_name: utf8, | |
| /// db_schema_name: utf8, | |
| /// table_name: utf8 not null, | |
| /// table_type: utf8 not null, | |
| /// \[optional\] table_schema: bytes not null (schema of the table as described in Schema.fbs::Schema, |
Current definition
/// The schema for GetTables without `table_schema` column
static GET_TABLES_SCHEMA_WITHOUT_TABLE_SCHEMA: Lazy<SchemaRef> = Lazy::new(|| {
Arc::new(Schema::new(vec![
Field::new("catalog_name", DataType::Utf8, false),
Field::new("db_schema_name", DataType::Utf8, false),
Field::new("table_name", DataType::Utf8, false),
Field::new("table_type", DataType::Utf8, false),
]))
});To Reproduce
- Schemas definition could be manually compared to other FlightSQL servers implementations, for example Cpp or Go.
- Run arrow-adbc tests pointing to FlightSQL-based server (see Additional context)
Expected behavior
Fields nullability is adjusted to fully correspond the protocol.
Additional context
Discovered when testing FlightSQL based server spiceai compatibility with arrow-adbc. There are tests failing due to schema mismatch, for example
Exception has occurred: CLR/Apache.Arrow.Adbc.C.CAdbcDriverImporter.ImportedAdbcException An exception of type 'Apache.Arrow.Adbc.C.CAdbcDriverImporter.ImportedAdbcException' occurred in Apache.Arrow.Adbc.dll but was not handled in user code: '[FlightSQL] Invalid schema returned for: expected schema: fields: 2 - catalog_name: type=utf8, nullable - db_schema_name: type=utf8, got schema: fields: 2 - catalog_name: type=utf8 - db_schema_name: type=utf8'