Skip to content

FlightSQL "GetDbSchemas" and "GetTables" schemas do not fully match the protocol #7637

@sgrebnov

Description

@sgrebnov

Describe the bug

The FlightSQL schema definition for CommandGetDbSchemas and CommandGetTables responses does not fully correspond to the protocol due to missing nullability information

CommandGetDbSchemas

catalog_name is expected to be nullable:

/// The returned Arrow schema will be:
/// <
/// catalog_name: utf8,
/// db_schema_name: utf8 not null
/// >
/// The returned data should be ordered by catalog_name, then db_schema_name.
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct CommandGetDbSchemas {

Current definition

/// The schema for GetDbSchemas
static GET_DB_SCHEMAS_SCHEMA: Lazy<SchemaRef> = Lazy::new(|| {
    Arc::new(Schema::new(vec![
        Field::new("catalog_name", DataType::Utf8, false),
        Field::new("db_schema_name", DataType::Utf8, false),
    ]))
});

CommandGetTables

catalog_name and db_schema_name are expected to be nullable:

/// The returned Arrow schema will be:
/// <
/// catalog_name: utf8,
/// db_schema_name: utf8,
/// table_name: utf8 not null,
/// table_type: utf8 not null,
/// \[optional\] table_schema: bytes not null (schema of the table as described in Schema.fbs::Schema,

Current definition

/// The schema for GetTables without `table_schema` column
static GET_TABLES_SCHEMA_WITHOUT_TABLE_SCHEMA: Lazy<SchemaRef> = Lazy::new(|| {
    Arc::new(Schema::new(vec![
        Field::new("catalog_name", DataType::Utf8, false),
        Field::new("db_schema_name", DataType::Utf8, false),
        Field::new("table_name", DataType::Utf8, false),
        Field::new("table_type", DataType::Utf8, false),
    ]))
});

To Reproduce

  1. Schemas definition could be manually compared to other FlightSQL servers implementations, for example Cpp or Go.

https://github.com/apache/arrow/blob/0e5249b0e8f060f8ee5136579bfeed8c92d0d1ab/cpp/src/arrow/flight/sql/server.cc#L1282-L1286

  1. Run arrow-adbc tests pointing to FlightSQL-based server (see Additional context)

Expected behavior

Fields nullability is adjusted to fully correspond the protocol.

Additional context

Discovered when testing FlightSQL based server spiceai compatibility with arrow-adbc. There are tests failing due to schema mismatch, for example

Exception has occurred: CLR/Apache.Arrow.Adbc.C.CAdbcDriverImporter.ImportedAdbcException An exception of type 'Apache.Arrow.Adbc.C.CAdbcDriverImporter.ImportedAdbcException' occurred in Apache.Arrow.Adbc.dll but was not handled in user code: '[FlightSQL] Invalid schema returned for: expected schema: fields: 2 - catalog_name: type=utf8, nullable - db_schema_name: type=utf8, got schema: fields: 2 - catalog_name: type=utf8 - db_schema_name: type=utf8'

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow cratearrow-flightChanges to the arrow-flight cratebug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions