[ty] Improve consistency and quality of diagnostics relating to invalid type forms by AlexWaygood · Pull Request #24325 · astral-sh/ruff

AlexWaygood · 2026-03-31T10:58:32Z

Summary

Currently this causes us to emit byte-string-type-annotation:

x: b"foo"

but this causes us to emit a different error code (invalid-type-form):

x: list[b"foo"]

This is pretty inconsistent. The cause is that we try to handle these AST nodes in both infer_annotation_expression and infer_type_expression, and the two code paths became inconsistent with each other. This PR adjusts our logic in infer_annotation_expression to just fallthrough to infer_type_expression for these AST nodes, and has both of the above examples use the invalid-type-form error code.

The exact same issue exists for our fstring-type-annotation rule, as well; this is also fixed in this PR. The error codes fstring-type-annotation and byte-string-type-annotation are now unused, so they are removed wholesale.

Since I was touching the relevant part of the code anyway, I also added some more subdiagnostic hints for some of these diagnostics as part of this PR.

Test Plan

Mdtests and snapshots.

astral-sh-bot · 2026-03-31T11:00:17Z

Typing conformance results

The percentage of diagnostics emitted that were expected errors held steady at 86.76%. The percentage of expected errors that received a diagnostic held steady at 81.53%. The number of fully passing files held steady at 70/132.

Summary

How are test cases classified?

Each test case represents one expected error annotation or a group of annotations sharing a tag. Counts are per test case, not per diagnostic — multiple diagnostics on the same line count as one. Required annotations (E) are true positives when ty flags the expected location and false negatives when it does not. Optional annotations (E?) are true positives when flagged but true negatives (not false negatives) when not. Tagged annotations (E[tag]) require ty to flag exactly one of the tagged lines; tagged multi-annotations (E[tag+]) allow any number up to the tag count. Flagging unexpected locations counts as a false positive.

Metric	Old	New	Diff
True Positives	865	865	+0
False Positives	132	132	+0
False Negatives	196	196	+0
Total Diagnostics	1050	1050	+0
Precision	86.76%	86.76%	+0.00%
Recall	81.53%	81.53%	+0.00%
Passing Files	70/132	70/132	+0

True positives changed (16)

16 diagnostics

Test case	Diff
aliases_type_statement.py:46	-error[invalid-type-form] Boolean literals are not allowed in this context in a type expression +error[invalid-type-form] Boolean literals are not allowed in this context in a type expression: Did you mean `typing.Literal[True]`?
aliases_type_statement.py:47	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
aliases_typealiastype.py:61	-error[invalid-type-form] Boolean literals are not allowed in this context in a type expression +error[invalid-type-form] Boolean literals are not allowed in this context in a type expression: Did you mean `typing.Literal[True]`?
aliases_typealiastype.py:62	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
annotations_forward_refs.py:50	-error[invalid-type-form] Boolean literals are not allowed in this context in a type expression +error[invalid-type-form] Boolean literals are not allowed in this context in a type expression: Did you mean `typing.Literal[True]`?
annotations_forward_refs.py:51	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
annotations_forward_refs.py:54	-error[fstring-type-annotation] Type expressions cannot use f-strings +error[invalid-type-form] F-strings are not allowed in type expressions
annotations_typeexpr.py:101	-error[fstring-type-annotation] Type expressions cannot use f-strings +error[invalid-type-form] F-strings are not allowed in type expressions
annotations_typeexpr.py:97	-error[invalid-type-form] Boolean literals are not allowed in this context in a type expression +error[invalid-type-form] Boolean literals are not allowed in this context in a type expression: Did you mean `typing.Literal[True]`?
annotations_typeexpr.py:98	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
classes_classvar.py:39	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[3]`?
directives_cast.py:16	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
generics_syntax_declarations.py:75	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[3]`?
qualifiers_annotated.py:46	-error[invalid-type-form] Boolean literals are not allowed in this context in a type expression +error[invalid-type-form] Boolean literals are not allowed in this context in a type expression: Did you mean `typing.Literal[True]`?
qualifiers_annotated.py:47	-error[invalid-type-form] Int literals are not allowed in this context in a type expression +error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[1]`?
qualifiers_annotated.py:49	-error[fstring-type-annotation] Type expressions cannot use f-strings +error[invalid-type-form] F-strings are not allowed in type expressions

astral-sh-bot · 2026-03-31T11:01:00Z

Memory usage report

Memory usage unchanged ✅

astral-sh-bot · 2026-03-31T11:03:12Z

`ecosystem-analyzer` results

Lint rule	Added	Removed	Changed
`invalid-await`	40	0	0
`escape-character-in-forward-annotation`	0	0	7
`invalid-argument-type`	0	1	0
`invalid-return-type`	1	0	0
`invalid-type-form`	0	0	1
Total	41	1	8

Changes in flaky projects detected. Raw diff output excludes flaky projects; see the HTML report for details.

Raw diff:

egglog-python (https://github.com/egraphs-good/egglog-python)
- python/egglog/exp/array_api.py:2878:35 error[invalid-argument-type] Argument to function `multiset_flat_map` is incorrect: Expected `(MultiSet[Value], /) -> MultiSet[MultiSet[Value]]`, found `UnstableFn`

jinja (https://github.com/pallets/jinja)
- src/jinja2/defaults.py:24:19 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ src/jinja2/defaults.py:24:19 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions
- src/jinja2/environment.py:307:27 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ src/jinja2/environment.py:307:27 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions
- src/jinja2/environment.py:399:27 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ src/jinja2/environment.py:399:27 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions
- src/jinja2/environment.py:1177:27 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ src/jinja2/environment.py:1177:27 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions

spack (https://github.com/spack/spack)
- lib/spack/spack/vendor/jinja2/defaults.py:24:19 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ lib/spack/spack/vendor/jinja2/defaults.py:24:19 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions
- lib/spack/spack/vendor/jinja2/environment.py:303:27 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ lib/spack/spack/vendor/jinja2/environment.py:303:27 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions
- lib/spack/spack/vendor/jinja2/environment.py:1161:27 error[escape-character-in-forward-annotation] Type expressions cannot contain escape characters
+ lib/spack/spack/vendor/jinja2/environment.py:1161:27 error[escape-character-in-forward-annotation] Escape characters are not allowed in type expressions

sympy (https://github.com/sympy/sympy)
- sympy/vector/implicitregion.py:130:49 error[invalid-type-form] Int literals are not allowed in this context in a type expression
+ sympy/vector/implicitregion.py:130:49 error[invalid-type-form] Int literals are not allowed in this context in a type expression: Did you mean `typing.Literal[0]`?

Full report with detailed diff (timing results)

carljm · 2026-03-31T14:55:14Z

It definitely seems to me like byte-string-type-annotation and fstring-type-annotation should both just be invalid-type-form -- feels very inconsistent to have those two split out as separate diagnostic codes, when lots of other similar cases are just invalid-type-form.

We are still in beta; there will never be a better time in future for error code changes.

AlexWaygood · 2026-03-31T15:00:03Z

Okay, I can just get rid of those error codes!

AlexWaygood · 2026-04-01T16:40:23Z

The ecosystem report shows the new "Did you mean Literal[0]?" hint misfiring in one location: https://github.com/sympy/sympy/blob/1e427cc30abbccbb91d6df2aadf4e098868f75da/sympy/vector/implicitregion.py#L130. It seems pretty clear to me that we shouldn't be parsing the 0 there as a type expression anyway (and I suspect that what sympy meant to write there was list(self.singular_points())[0] rather than list[self.singular_points()][0]). I initially tried to fix that in #24326, but @carljm pointed out that my technique for doing so was kinda hacky and dubious. I'll continue to look into it, but for now I think a single misfire in the ecosystem isn't really that bad.

charliermarsh · 2026-04-02T13:03:47Z

crates/ty_python_semantic/resources/mdtest/annotations/invalid.md

        o: bar(),  # error: [invalid-type-form] "Function calls are not allowed in type expressions"
        # error: [unsupported-operator]
-        # error: [invalid-type-form] "F-strings are not allowed in type expressions"
+        # error: [invalid-type-form] "Type expressions cannot use f-strings"


I might say "include" instead of "use", but it looks like this is consistent with our existing diagnostics elsewhere, so feel free to ignore.

I think there was some inconsistency here -- thank you! Tried to clean it up

charliermarsh · 2026-04-02T13:04:08Z

crates/ty_python_semantic/resources/mdtest/annotations/string.md

+    c: f"int",
+    # error: [invalid-type-form] "Type expressions cannot use f-strings"
+    d: list[f"int"],
+    # error: [invalid-type-form] "Bytes literals are not allowed in this context in a type expression"


Does the message differ here because bytes literals are sometimes allowed in type expressions (e.g., within Literal)?

Yup, exactly. It's not true to say they're always invalid in type expressions -- they're just only valid in certain specific contexts

charliermarsh · 2026-04-02T13:04:26Z

...ests_for_invalid_ty…_-_Diagnostics_for_comm…_-_Dict-literal_or_set-…_(15737b0beb194b0e).snap

+  |
+1 | def _(
+2 |     x: {int: str},  # error: [invalid-type-form]
+  |        ^^^^^^^^^^ Did you mean `dict[int, str]`?


…id type forms

* main: Add a "release-gate" step to the release workflow (#24365) Disallow starred expressions as values of starred expressions (#24280) [`pyupgrade`] Ignore strings with string-only escapes (`UP012`) (#16058) [ty] Improve consistency and quality of diagnostics relating to invalid type forms (#24325) [flake8-type-checking] Clarify import cycle wording for TC001/TC002/TC003 (#24322) [`flake8-errmsg`] Avoid shadowing existing `msg` in fix for `EM101` (#24363) `RUF072`: skip formfeeds on dedent (#24308) Replace unmaintained `unic-ucd-category` crate with `icu_properties` (#24344) [ty] Replace markdown hard line breaks in snapshot tests (#24361) [ty] Move snapshot for code action test with trailing whitespace to external file (#24359)

AlexWaygood added ty Multi-file analysis & type inference diagnostics Related to reporting of diagnostics. labels Mar 31, 2026

AlexWaygood force-pushed the alex/invalid-expr-messages branch 2 times, most recently from 2755643 to 7c45fad Compare March 31, 2026 12:17

AlexWaygood changed the base branch from main to alex/cascading-type-expr-diagnostics March 31, 2026 12:18

AlexWaygood closed this Mar 31, 2026

AlexWaygood reopened this Mar 31, 2026

AlexWaygood force-pushed the alex/invalid-expr-messages branch from 7c45fad to e7e60af Compare March 31, 2026 12:59

AlexWaygood marked this pull request as ready for review March 31, 2026 13:20

AlexWaygood requested review from carljm, dcreager, ibraheemdev and sharkdp as code owners March 31, 2026 13:20

astral-sh-bot bot assigned sharkdp Mar 31, 2026

carljm removed their request for review March 31, 2026 14:55

AlexWaygood marked this pull request as draft March 31, 2026 22:06

AlexWaygood force-pushed the alex/cascading-type-expr-diagnostics branch 2 times, most recently from a18198c to 9ddda47 Compare April 1, 2026 10:56

Base automatically changed from alex/cascading-type-expr-diagnostics to main April 1, 2026 11:09

AlexWaygood force-pushed the alex/invalid-expr-messages branch 2 times, most recently from 28b482c to a19616d Compare April 1, 2026 16:38

AlexWaygood marked this pull request as ready for review April 1, 2026 16:40

AlexWaygood requested a review from MichaReiser as a code owner April 1, 2026 16:40

AlexWaygood force-pushed the alex/invalid-expr-messages branch from a19616d to b89cadd Compare April 1, 2026 17:31

AlexWaygood force-pushed the alex/invalid-expr-messages branch 2 times, most recently from 993948e to 8c3251f Compare April 1, 2026 20:37

charliermarsh reviewed Apr 2, 2026

View reviewed changes

charliermarsh approved these changes Apr 2, 2026

View reviewed changes

[ty] Improve consistency and quality of diagnostics relating to inval…

246831b

…id type forms

AlexWaygood force-pushed the alex/invalid-expr-messages branch from 8c3251f to 246831b Compare April 2, 2026 14:11

AlexWaygood enabled auto-merge (squash) April 2, 2026 14:12

AlexWaygood merged commit 34d54b6 into main Apr 2, 2026
50 checks passed

AlexWaygood deleted the alex/invalid-expr-messages branch April 2, 2026 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ty] Improve consistency and quality of diagnostics relating to invalid type forms#24325

[ty] Improve consistency and quality of diagnostics relating to invalid type forms#24325
AlexWaygood merged 1 commit intomainfrom
alex/invalid-expr-messages

AlexWaygood commented Mar 31, 2026 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

carljm commented Mar 31, 2026

Uh oh!

AlexWaygood commented Mar 31, 2026

Uh oh!

AlexWaygood commented Apr 1, 2026

Uh oh!

charliermarsh Apr 2, 2026

Uh oh!

AlexWaygood Apr 2, 2026

Uh oh!

charliermarsh Apr 2, 2026

Uh oh!

AlexWaygood Apr 2, 2026

Uh oh!

charliermarsh Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

AlexWaygood commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Typing conformance results

Summary

True positives changed (16)

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Memory usage report

Uh oh!

astral-sh-bot bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

carljm commented Mar 31, 2026

Uh oh!

AlexWaygood commented Mar 31, 2026

Uh oh!

AlexWaygood commented Apr 1, 2026

Uh oh!

charliermarsh Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

charliermarsh Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

charliermarsh Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlexWaygood commented Mar 31, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 31, 2026 •

edited

Loading

`ecosystem-analyzer` results