-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Define format schema via query #21096
Copy link
Copy link
Closed
Labels
comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).duplicatefeature
Description
Use case
Before we use KafkaEngine to consume Protobuf data from kafka, we must define the format schema and place the *.proto file in the specified directroy (format_schema_path). It's easy for single node. But when we have a cluster, we need to put the schema file on every node of the cluster manually. Of course, this can be implemented by external tools or service, but it's still inconvenient and it might increased workload for developers.
It will be easier if we can define the format schema via query.
Describe the solution you'd like
- Support the schema definition query like below
CREARE SCHEMA IF NOT EXISTS `schema_file`.`type_name` on cluster test_cluster
(
f1 String,
f2 UInt32,
f3 Array(UInt32)
)
Format ProtobufAfter executing the query , schema_file.proto file will be create in the specified directory, and the message type type_name will be define in the file.
- Add system table for created schema so that we can check the schema by query.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).duplicatefeature
