Skip to content

Define format schema via query #21096

@BigRantLing

Description

@BigRantLing

Use case
Before we use KafkaEngine to consume Protobuf data from kafka, we must define the format schema and place the *.proto file in the specified directroy (format_schema_path). It's easy for single node. But when we have a cluster, we need to put the schema file on every node of the cluster manually. Of course, this can be implemented by external tools or service, but it's still inconvenient and it might increased workload for developers.
It will be easier if we can define the format schema via query.

Describe the solution you'd like

  1. Support the schema definition query like below
CREARE SCHEMA IF NOT EXISTS `schema_file`.`type_name` on cluster test_cluster
(
   f1 String, 
   f2 UInt32,
   f3 Array(UInt32) 
) 
Format Protobuf

After executing the query , schema_file.proto file will be create in the specified directory, and the message type type_name will be define in the file.

  1. Add system table for created schema so that we can check the schema by query.

Demo
We impleted a simple demo for Protobuf like below.
mP3Qqn

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).duplicatefeature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions