Skip to content

[ty] Dataclass field converters#23088

Draft
sharkdp wants to merge 2 commits intomainfrom
david/converter
Draft

[ty] Dataclass field converters#23088
sharkdp wants to merge 2 commits intomainfrom
david/converter

Conversation

@sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Feb 5, 2026

Summary

Adds support for dataclass field converters.

closes astral-sh/ty#972

Ecosystem impact

Lots of removed false positives on attrs, home-assistant/core, and trio.

Typing conformance results

With this PR, we pass almost all tests in dataclasses_transform_converter.py. The remaining problem in this test suite is not related to dataclasses or dataclass converters, but rather to a limitation in our generics solver (we don't understand the call to field, and therefore don't recognize the converter function).

Test Plan

New Markdown tests

@sharkdp sharkdp added ty Multi-file analysis & type inference ecosystem-analyzer labels Feb 5, 2026
@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 5, 2026

Typing conformance results improved 🎉

The percentage of diagnostics emitted that were expected errors increased from 85.05% to 85.42%. The percentage of expected errors that received a diagnostic increased from 78.05% to 78.15%. The number of fully passing files held steady at 63/132.

Summary

How are test cases classified?

Each test case represents one expected error annotation or a group of annotations sharing a tag. Counts are per test case, not per diagnostic — multiple diagnostics on the same line count as one. Required annotations (E) are true positives when ty flags the expected location and false negatives when it does not. Optional annotations (E?) are true positives when flagged but true negatives (not false negatives) when not. Tagged annotations (E[tag]) require ty to flag exactly one of the tagged lines; tagged multi-annotations (E[tag+]) allow any number up to the tag count. Flagging unexpected locations counts as a false positive.

Metric Old New Diff Outcome
True Positives 825 826 +1 ⏫ (✅)
False Positives 145 141 -4 ⏬ (✅)
False Negatives 232 231 -1 ⏬ (✅)
Total Diagnostics 1047 1024 -23
Precision 85.05% 85.42% +0.37% ⏫ (✅)
Recall 78.05% 78.15% +0.09% ⏫ (✅)
Passing Files 63/132 63/132 +0

Test file breakdown

1 file altered
File True Positives False Positives False Negatives Status
dataclasses_transform_converter.py 9 (+1) ✅ 2 (-4) ✅ 0 (-1) ✅ 📈 Improving
Total (all files) 826 (+1) ✅ 141 (-4) ✅ 231 (-1) ✅ 63/132

True positives added (1)

1 diagnostic
Test case Diff

dataclasses_transform_converter.py:118

+error[invalid-assignment] Object of type `Literal[1]` is not assignable to attribute `field0` of type `str`

False positives removed (4)

4 diagnostics
Test case Diff

dataclasses_transform_converter.py:112

-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f0"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f2"]`
-error[invalid-argument-type] Argument is incorrect: Expected `ConverterClass`, found `Literal[b"f6"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `list[Unknown]`

dataclasses_transform_converter.py:114

-error[invalid-assignment] Object of type `Literal["f1"]` is not assignable to attribute `field0` of type `int`

dataclasses_transform_converter.py:115

-error[invalid-assignment] Object of type `Literal["f6"]` is not assignable to attribute `field3` of type `ConverterClass`

dataclasses_transform_converter.py:116

-error[invalid-assignment] Object of type `Literal[b"f6"]` is not assignable to attribute `field3` of type `ConverterClass`

True positives changed (4)

4 diagnostics
Test case Diff

dataclasses_transform_converter.py:107

-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f2"]`
-error[invalid-argument-type] Argument is incorrect: Expected `ConverterClass`, found `Literal[b"f3"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `list[Unknown]`
+error[invalid-argument-type] Argument is incorrect: Expected `str`, found `Literal[1]`

dataclasses_transform_converter.py:108

-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f0"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f2"]`
-error[invalid-argument-type] Argument is incorrect: Expected `ConverterClass`, found `Literal[1]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `list[Unknown]`
+error[invalid-argument-type] Argument is incorrect: Expected `str | bytes`, found `Literal[1]`

dataclasses_transform_converter.py:109

-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f0"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f2"]`
-error[invalid-argument-type] Argument is incorrect: Expected `ConverterClass`, found `Literal["f3"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `complex`
+error[invalid-argument-type] Argument is incorrect: Expected `str | list[str]`, found `complex`

dataclasses_transform_converter.py:119

-error[invalid-assignment] Object of type `Literal[1]` is not assignable to attribute `field3` of type `ConverterClass`
+error[invalid-assignment] Object of type `Literal[1]` is not assignable to attribute `field3` of type `str | bytes`

False positives changed (1)

1 diagnostic
Test case Diff

dataclasses_transform_converter.py:121

-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f0"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["f2"]`
-error[invalid-argument-type] Argument is incorrect: Expected `ConverterClass`, found `Literal["f6"]`
-error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["1"]`
-error[invalid-argument-type] Argument is incorrect: Expected `dict[str, str]`, found `tuple[tuple[Literal["a"], Literal["1"]], tuple[Literal["b"], Literal["2"]]]`
+error[invalid-argument-type] Argument is incorrect: Expected `dict[str, str]`, found `tuple[tuple[Literal["a"], Literal["1"]], tuple[Literal["b"], Literal["2"]]]`

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 5, 2026

mypy_primer results

Changes were detected when running on open source projects
attrs (https://github.com/python-attrs/attrs)
- tests/dataclass_transform_example.py:21:13: info[revealed-type] Revealed type: `(self: DefineConverter, with_converter: int) -> None`
+ tests/dataclass_transform_example.py:21:13: info[revealed-type] Revealed type: `(self: DefineConverter, with_converter: str | Buffer | SupportsInt | SupportsIndex | SupportsTrunc) -> None`
- tests/dataclass_transform_example.py:23:17: error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal[b"42"]`
- tests/test_hooks.py:70:15: error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["3"]`
- tests/test_next_gen.py:380:14: error[unresolved-attribute] Object of type `dataclasses.Field` has no attribute `validator`
+ tests/test_next_gen.py:380:14: error[unresolved-attribute] Object of type `dataclasses.Field[int]` has no attribute `validator`
- tests/test_next_gen.py:388:9: error[invalid-assignment] Object of type `Literal["11"]` is not assignable to attribute `x` of type `int`
- tests/test_next_gen.py:394:13: error[invalid-assignment] Object of type `Literal["9"]` is not assignable to attribute `x` of type `int`
- typing-examples/mypy.py:181:13: error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["on"]`
- typing-examples/mypy.py:182:13: error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["yes"]`
- typing-examples/mypy.py:185:13: error[invalid-argument-type] Argument is incorrect: Expected `int`, found `Literal["n"]`
- Found 671 diagnostics
+ Found 664 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- pydantic/_internal/_core_metadata.py:87:54: error[invalid-assignment] Invalid assignment to key "pydantic_js_extra" with declared type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | ((dict[str, Divergent], type[Any], /) -> None)` on TypedDict `CoreMetadata`: value of type `dict[object, object]`
+ pydantic/_internal/_core_metadata.py:87:54: error[invalid-assignment] Invalid assignment to key "pydantic_js_extra" with declared type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | ((dict[str, int | float | str | ... omitted 3 union elements], type[Any], /) -> None)` on TypedDict `CoreMetadata`: value of type `dict[object, object]`
- pydantic/fields.py:949:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:949:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:989:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:989:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1032:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1032:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1072:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1072:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1115:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1115:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1154:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1154:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1194:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1194:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1573:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `dict[str, Divergent] | dict[Never, Never] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1573:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `dict[str, int | float | str | ... omitted 3 union elements] | dict[Never, Never] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

trio (https://github.com/python-trio/trio)
- src/trio/_tests/test_highlevel_open_tcp_listeners.py:235:35: error[invalid-argument-type] Argument is incorrect: Expected `SocketKind`, found `int`
- Found 469 diagnostics
+ Found 468 diagnostics

core (https://github.com/home-assistant/core)
+ homeassistant/helpers/device_registry.py:327:32: error[invalid-assignment] Object of type `dataclasses.Field[set[_T@set]]` is not assignable to `set[str]`
+ homeassistant/helpers/device_registry.py:338:41: error[invalid-assignment] Object of type `dataclasses.Field[set[_T@set]]` is not assignable to `set[tuple[str, str]]`
+ homeassistant/helpers/device_registry.py:339:24: error[invalid-assignment] Object of type `dataclasses.Field[set[_T@set]]` is not assignable to `set[str]`
- homeassistant/helpers/entity_registry.py:1244:13: error[invalid-argument-type] Argument is incorrect: Expected `str`, found `None | str`
- homeassistant/helpers/entity_registry.py:1248:13: error[invalid-argument-type] Argument is incorrect: Expected `ReadOnlyEntityOptionsType`, found `Mapping[str, Mapping[str, Any]] | None`
- Found 12088 diagnostics
+ Found 12089 diagnostics

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 5, 2026

ecosystem-analyzer results

Lint rule Added Removed Changed
invalid-argument-type 0 8 0
invalid-assignment 3 2 0
unresolved-attribute 0 0 1
Total 3 10 1

Full report with detailed diff (timing results)

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 13, 2026

Memory usage report

Summary

Project Old New Diff Outcome
prefect 701.30MB 701.32MB +0.00% (21.95kB)
sphinx 265.18MB 265.19MB +0.00% (5.12kB)
trio 117.80MB 117.80MB +0.00% (912.00B)
flake8 47.90MB 47.91MB +0.00% (896.00B)

Significant changes

Click to expand detailed breakdown

prefect

Name Old New Diff Outcome
StaticClassLiteral<'db>::fields_ 96.19kB 115.91kB +20.50% (19.72kB)
infer_scope_types_impl 52.62MB 52.62MB +0.00% (2.11kB)
FieldInstance 384.00B 512.00B +33.33% (128.00B)

sphinx

Name Old New Diff Outcome
StaticClassLiteral<'db>::fields_ 18.64kB 22.86kB +22.63% (4.22kB)
infer_scope_types_impl 15.59MB 15.59MB +0.01% (888.00B)
FieldInstance 96.00B 128.00B +33.33% (32.00B)

trio

Name Old New Diff Outcome
infer_scope_types_impl 4.79MB 4.79MB +0.02% (912.00B)

flake8

Name Old New Diff Outcome
StaticClassLiteral<'db>::fields_ 4.11kB 4.98kB +21.31% (896.00B)

@sharkdp sharkdp force-pushed the david/converter branch 2 times, most recently from 547d926 to 5c43bb1 Compare February 27, 2026 13:20
// For dataclass fields with converters, the write type is the converter's
// input type, not the field's declared type.
let effective_write_type = |attr_ty: Type<'db>| -> Type<'db> {
if let Type::NominalInstance(instance) = object_ty {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we desconstruct unions / intersections at a higher level, so we can narrowly match on NominalInstance here and still validate writes to unions of field types with converters involved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need a .resolve_type_alias(db)? (And a test for writing to something typed as an alias to a dataclass instance type).

Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!


@my_model
class WithClassConverter:
a: int = field(converter=PermissiveNumber)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by the int annotation here. PermissiveNumber is a callable which accepts an int | str and returns a PermissiveNumber instance. So I would expect a: PermissiveNumber here, which seems to match the conformance suite expectations: https://github.com/python/typing/blob/main/conformance/tests/dataclasses_transform_converter.py#L102

It seems like this line as written should be an error, and it is an error in pyright/pyrefly/mypy.

a: str is also not an error here in ty -- it seems like we aren't validating the type of class converter fields at all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I would expect a: PermissiveNumber here

Yes, absolutely. I think I rewrote the test too often and then overlooked this.

it seems like we aren't validating the type of class converter fields at all?

Yes, thanks for catching this! This is an artifact of the fact that we treat field(…) return types in a special way, because they're often annotated incorrectly. Looking into it.

if !matches!(field_policy, CodeGeneratorKind::DataclassLike(_)) {
return None;
}
let fields = self.own_fields(db, None, field_policy);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this do the right thing for writes to a subclass, where the converter-using field is inherited from a base class?

}

if let Some(converter_ty) = converter_input_type {
field_ty = converter_ty;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this method is currently also used for synthesizing __replace__ signature, but I think __replace__ should always use the raw field type, not the converter input type.

// For dataclass fields with converters, the write type is the converter's
// input type, not the field's declared type.
let effective_write_type = |attr_ty: Type<'db>| -> Type<'db> {
if let Type::NominalInstance(instance) = object_ty {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need a .resolve_type_alias(db)? (And a test for writing to something typed as an alias to a dataclass instance type).

Comment on lines +1145 to +1147
let mut input_types = UnionBuilder::new(db);
let mut found_any = false;
for binding in converter_ty.bindings(db).iter_flat() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of iter_flat means that we collapse the union/intersection structure of converter_ty.

If it is an intersection of callables, I think unioning the discovered first-parameter types is actually correct (an intersection of callables is similar to overloads, in that the callable accepts all of those types.)

But if converter_ty is a union of callables, then I think technically we should build an intersection of their first parameter types? If the converter either accepts A or B, but we don't know which, then only A & B is safe to provide to the field.

(Totally open to saying we don't need to care about this, but it's at least worth a comment, I think. Pyright and mypy seem to just fail in this union-of-callables case and ignore the converter entirely; pyrefly does the same as this PR and uses the union of the first-argument types.)

@sharkdp sharkdp marked this pull request as draft March 10, 2026 13:55
@rolfmorel
Copy link

As there's progress here, I thought to cross-post astral-sh/ty#1327 (comment):

@sharkdp, great that you're working on this!

Regarding getting converter support landed, it would be great to know where you/ty stand regarding the following question:

python/typing#2189: Does converter also support default argument in field specifiers of dataclass_transform?

CC: @PragmaTwice

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ecosystem-analyzer ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for dataclass_transform converters

3 participants