Do we know what was the actual problem here? Is there a specific feature and/or bug fix in pydantic 2.12 that we are using? I had a quick look at the release notes, but nothing stood out to me as particularly relevant for us

Do we know what was the actual problem here? Is there a specific feature and/or bug fix in pydantic 2.12 that we are using? I had a quick look at the release notes, but nothing stood out to me as particularly relevant for us

That's a good question... honestly I haven't had time to investigate deeper

I know that it was broken with pydantic==2.10.6 and pydantic_core==2.27.2 and it works with pydantic==2.12.5 and pydantic_core==2.41.5.

With the old version it was throwing these errors:

ValidationError: 2 validation errors for SparkDataFrameDataSource connection Field required [type=missing, input_value={'name': 'ias_bug_ds', 'c...schema_': 'ias_bug_ds'}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.8/v/missing connection_properties Extra inputs are not permitted [type=extra_forbidden, input_value={'spark_session': <pyspar...'schema_': 'ias_bug_ds'}, input_type=dict] For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

I don't really have a hypothesis as to why this was happening and I think I should probably not spend more time on this

Took a look at what caused this. We're using a feature that was introduced in pydantic 2.11, so naturally it breaks with older versions.

To be more specific, we configure DataSourceBase as follows:

class DataSourceBase(BaseModel, abc.ABC):
    model_config = ConfigDict(
        frozen=True,
        extra="forbid",
        validate_by_name=True,  # Allow to use both field names and aliases when populating from dict
    )
    connection_properties: DataSourceConnectionProperties = Field(
        ..., alias="connection", description="Data source connection details"
    )

We define connection_properties and alias it as connection. The validate_by_name flag (new in pydantic 2.11) allows initializing the field using the non-aliased key, i.e. connection_properties. Versions before that ignore this flag and don't recognize the field. Add in the fact we have extra="forbid", and it will explicitly error out on receiving connection_properties as a field.

That is exactly what pydantic reports in the output @paulteehan shared:

ValidationError: 2 validation errors for SparkDataFrameDataSource connection Field required [type=missing, input_value={'name': 'ias_bug_ds', 'c...schema_': 'ias_bug_ds'}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.8/v/missing connection_properties Extra inputs are not permitted [type=extra_forbidden, input_value={'spark_session': <pyspar...'schema_': 'ias_bug_ds'}, input_type=dict] For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

Formatted for readability:

ValidationError: 2 validation errors for SparkDataFrameDataSource
* connection
  * Field required [type=missing, input_value={'name': 'ias_bug_ds', 'c...schema_': 'ias_bug_ds'}}, input_type=dict]
  * For further information visit https://errors.pydantic.dev/2.8/v/missing
* connection_properties
  * Extra inputs are not permitted [type=extra_forbidden, input_value={'spark_session': <pyspar...'schema_': 'ias_bug_ds'}, input_type=dict]
  * For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden

i.e. connection_properties was provided when connection is required.

In summary: this isn't a bug in the databricks runtimes nor pydantic. It's just us messing up our versioning.

Updated to pydantic>=2.11, as that is the lowest version we actually need

Nicely done @mivds !!! Really appreciate the deep dive, it's great to have an actual explanation for this

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud