Skip to content

Avro Schema Evolution Support  #11986

@SergeyIBM47

Description

@SergeyIBM47

I use AvroConfulent data format with schema registry to consume Kafka events to clickhouse.

  • сurrently, Avro schemas are cached once resolved
  • after evolving Avro schema(add fields) clickhouse doesn't fetch new schema and failed with an error on SELECT raws from Kafka table
  • recreate Kafka table, change format_avro_schema_registry_url and restart clickhouse server doesn't refresh the cache
  • if I create the same Table on another Kafka topic with the same format everything works correctly, and I guess that schema cache works by schema name

As I understand these lines of code and comments in PR, fetch a new Avro schema should work automatically:
https://github.com/ClickHouse/ClickHouse/pull/8953/files/927e572d39432d22ae96e087674a5124c6e2931b#diff-4c0061c616a674630b9e2e74706e6255R61
#8571

AvroConfluent only caches schemas per an instance of an InputFormat. This means SchemaRegistry will be queried each time a batch from Kafka is processed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions