Skip to content

New types can break older Iceberg agents #10775

@jacobmarble

Description

@jacobmarble

After #9008 is merged, it becomes possible for an Iceberg agent to create schemas with timestamp_ns columns. The behavior of Iceberg agents not aware of this new type is undefined.

A simple solution: new types require format-version=3


Originally posted by @nastra in #9008 (comment)

Given that this type is for v3, wouldn't a v2 writer just blindly write this new type if it's being used anywhere and thus break forward compability?

@nastra I don't think I understand your question, would you mind sharing an example?

It seems to me that any writer that doesn't recognize a particular type (in this case timestamp_ns or timestamptz_ns) should fail when it reads the schema in the table metadata. How does the writer "blindly" write to an unimplemented type?

@jacobmarble currently nothing is preventing this new type to be written with a V2 writer, meaning that one could use this new type in a schema and write it with a V2 writer. Now an older Iceberg client tries to read this using a V2 reader and will fail.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions