Skip to content

Removing then adding the same partition in update_spec fails #1131

@lkindere

Description

@lkindere

Apache Iceberg version

0.7.1 (latest release)

Please describe the bug 🐞

When adding a field to partition spec, if the field previously existed but was deleted it will fail when not explictly defining a column name:

For example this would fail:
table.update_spec().add_identity("VendorID").commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_identity("VendorID").commit()

This would fail:
table.update_spec().add_field("VendorID", IdentityTransform()).commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_field("VendorID", IdentityTransform()).commit()

This would not fail when defining the partition field name:
table.update_spec().add_field("VendorID", IdentityTransform(), "VendorID").commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_field("VendorID", IdentityTransform(), "VendorID").commit()

Example trace:
Traceback (most recent call last):
File "C:\Users\a\PycharmProjects\pythonProject\test.py", line 24, in
table.update_spec().add_identity("VendorID").commit()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init_.py", line 3682, in add_identity
return self.add_field(source_column_name, IdentityTransform(), None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init_.py", line 3657, in add_field
new_field = self.partition_field((bound_ref.field.field_id, transform), partition_field_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init
.py", line 3833, in _partition_field
return PartitionField(source_id, field_key[1], transform, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\partitioning.py", line 111, in init
super().init(**data)
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pydantic\main.py", line 193, in init
self.pydantic_validator.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for PartitionField
name
Field required [type=missing, input_value={'source-id': 1, 'field-i...m': IdentityTransform()}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.8/v/missing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions