Skip to content

[ty] filter out pre-loop bindings from loop headers#23536

Merged
oconnor663 merged 1 commit intomainfrom
jack/loop_header_bindings_in_loop_only
Mar 3, 2026
Merged

[ty] filter out pre-loop bindings from loop headers#23536
oconnor663 merged 1 commit intomainfrom
jack/loop_header_bindings_in_loop_only

Conversation

@oconnor663
Copy link
Contributor

@oconnor663 oconnor663 commented Feb 24, 2026

Reading through @mtshiba's #23520 (specifically if def != definition here) made it clear that the way I included pre-loop bindings in loop headers in #22794 is kind of gross. Apart from being fully redundant (loop headers don't shadow pre-existing bindings), it also means that loop headers tracks themselves, which presumably creates pointless cycles that need to be resolved. Since definition IDs are issued in source code order, it's relatively easy to filter out all the definitions that existed before the loop began, including the loop headers. Anecdotally this seems to give a ~10% speedup in isort, which is the repo that motivated #23520. I'll be curious to see if the ecosystem report shows changed diagnostics, but I'm not sure it will.

Before landing this:

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 24, 2026

Typing conformance results

No changes detected ✅

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 24, 2026

mypy_primer results

Changes were detected when running on open source projects
alerta (https://github.com/alerta/alerta)
- alerta/utils/api.py:123:48: error[invalid-assignment] Object of type `Unknown | None | Any | (tuple[Divergent, Divergent, Divergent, Divergent] & Alert) | (tuple[Divergent, str | Unknown, str | Unknown, int | Unknown] & Alert)` is not assignable to `Alert`
- alerta/utils/api.py:125:17: error[invalid-assignment] Too many values to unpack: Expected 3
- Found 627 diagnostics
+ Found 625 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- pydantic/_internal/_core_metadata.py:87:54: error[invalid-assignment] Invalid assignment to key "pydantic_js_extra" with declared type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | ((dict[str, int | float | str | ... omitted 3 union elements], type[Any], /) -> None)` on TypedDict `CoreMetadata`: value of type `dict[object, object]`
+ pydantic/_internal/_core_metadata.py:87:54: error[invalid-assignment] Invalid assignment to key "pydantic_js_extra" with declared type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | ((dict[str, Divergent], type[Any], /) -> None)` on TypedDict `CoreMetadata`: value of type `dict[object, object]`
- pydantic/fields.py:949:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:949:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:989:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:989:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1032:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1032:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1072:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1072:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1115:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1115:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1154:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1154:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1194:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
+ pydantic/fields.py:1194:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
- pydantic/fields.py:1573:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1573:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/deployments/runner.py:1017:70: error[unresolved-attribute] Attribute `__name__` is not defined on `(...) -> Any` in union `Unknown | ((...) -> Any)`
+ src/prefect/deployments/runner.py:1017:70: error[unresolved-attribute] Attribute `__name__` is not defined on `((...) -> Any) & ((*args: object, **kwargs: object) -> object)` in union `Unknown | (((...) -> Any) & ((*args: object, **kwargs: object) -> object))`
- src/prefect/flow_engine.py:1004:32: error[invalid-await] `Unknown | R@FlowRunEngine | Coroutine[Any, Any, R@FlowRunEngine]` is not awaitable
- src/prefect/flow_engine.py:1610:24: error[invalid-await] `Unknown | R@AsyncFlowRunEngine | Coroutine[Any, Any, R@AsyncFlowRunEngine]` is not awaitable
- src/prefect/flows.py:286:34: error[unresolved-attribute] Object of type `(**P@Flow) -> R@Flow` has no attribute `__name__`
+ src/prefect/flows.py:286:34: error[unresolved-attribute] Object of type `((**P@Flow) -> R@Flow) & ((*args: object, **kwargs: object) -> object)` has no attribute `__name__`
- src/prefect/flows.py:404:68: error[unresolved-attribute] Object of type `(**P@Flow) -> R@Flow` has no attribute `__name__`
+ src/prefect/flows.py:404:68: error[unresolved-attribute] Object of type `((**P@Flow) -> R@Flow) & ((*args: object, **kwargs: object) -> object)` has no attribute `__name__`
- Found 5802 diagnostics
+ Found 5800 diagnostics

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
- src/scikit_build_core/build/wheel.py:99:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 58 diagnostics
+ Found 57 diagnostics

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 24, 2026

Memory usage report

Summary

Project Old New Diff Outcome
trio 118.48MB 118.46MB -0.02% (22.90kB) ⬇️
flake8 48.12MB 48.07MB -0.09% (43.08kB) ⬇️
sphinx 267.28MB 266.74MB -0.20% (557.55kB) ⬇️
prefect 693.27MB 692.22MB -0.15% (1.05MB) ⬇️

Significant changes

Click to expand detailed breakdown

trio

Name Old New Diff Outcome
get_loop_header 82.12kB 72.88kB -11.25% (9.24kB) ⬇️
infer_definition_types 7.63MB 7.62MB -0.06% (4.52kB) ⬇️
all_negative_narrowing_constraints_for_expression 190.12kB 186.74kB -1.78% (3.38kB) ⬇️
loop_header_reachability 145.40kB 142.88kB -1.73% (2.52kB) ⬇️
all_narrowing_constraints_for_expression 612.63kB 610.16kB -0.40% (2.47kB) ⬇️
semantic_index 30.33MB 30.33MB -0.01% (1.77kB) ⬇️
is_redundant_with_impl::interned_arguments 572.09kB 573.03kB +0.17% (968.00B) ⬇️
is_redundant_with_impl 495.46kB 496.30kB +0.17% (864.00B) ⬇️
infer_expression_type_impl 1.49MB 1.49MB -0.04% (684.00B) ⬇️
infer_expression_types_impl 7.15MB 7.15MB -0.00% (208.00B) ⬇️
infer_scope_types_impl 4.79MB 4.79MB +0.00% (180.00B) ⬇️
Type<'db>::member_lookup_with_policy_::interned_arguments 875.98kB 875.88kB -0.01% (104.00B) ⬇️
IntersectionType<'db>::from_two_elements_::interned_arguments 63.34kB 63.42kB +0.14% (88.00B) ⬇️
Type<'db>::member_lookup_with_policy_ 1.71MB 1.71MB -0.00% (64.00B) ⬇️
place_table 76.76kB 76.70kB -0.08% (60.00B) ⬇️
... 2 more

flake8

Name Old New Diff Outcome
infer_expression_type_impl 170.94kB 161.00kB -5.82% (9.94kB) ⬇️
infer_definition_types 1.89MB 1.88MB -0.48% (9.34kB) ⬇️
infer_expression_types_impl 1.08MB 1.07MB -0.82% (9.02kB) ⬇️
loop_header_reachability 20.50kB 14.96kB -27.01% (5.54kB) ⬇️
all_negative_narrowing_constraints_for_expression 44.83kB 40.73kB -9.16% (4.11kB) ⬇️
all_narrowing_constraints_for_expression 87.96kB 84.08kB -4.41% (3.88kB) ⬇️
get_loop_header 16.17kB 13.96kB -13.67% (2.21kB) ⬇️
is_redundant_with_impl::interned_arguments 157.95kB 158.64kB +0.44% (704.00B) ⬇️
semantic_index 13.83MB 13.83MB -0.00% (384.00B) ⬇️
is_redundant_with_impl 149.98kB 150.35kB +0.25% (384.00B) ⬇️
UnionType 111.16kB 111.30kB +0.13% (144.00B) ⬇️
IntersectionType<'db>::from_two_elements_::interned_arguments 21.66kB 21.74kB +0.40% (88.00B) ⬇️
IntersectionType<'db>::from_two_elements_ 21.06kB 21.11kB +0.26% (56.00B) ⬇️
infer_scope_types_impl 1006.96kB 1006.95kB -0.00% (12.00B) ⬇️

sphinx

Name Old New Diff Outcome
infer_expression_type_impl 3.38MB 3.26MB -3.40% (117.51kB) ⬇️
infer_expression_types_impl 21.90MB 21.79MB -0.48% (108.57kB) ⬇️
infer_definition_types 24.35MB 24.26MB -0.39% (97.80kB) ⬇️
get_loop_header 401.29kB 330.71kB -17.59% (70.58kB) ⬇️
semantic_index 62.54MB 62.49MB -0.07% (47.34kB) ⬇️
loop_header_reachability 436.10kB 395.18kB -9.38% (40.93kB) ⬇️
all_narrowing_constraints_for_expression 2.39MB 2.36MB -1.23% (29.98kB) ⬇️
all_negative_narrowing_constraints_for_expression 1.04MB 1.02MB -1.54% (16.41kB) ⬇️
is_redundant_with_impl::interned_arguments 2.14MB 2.14MB -0.21% (4.64kB) ⬇️
StaticClassLiteral<'db>::try_mro_ 2.18MB 2.18MB -0.16% (3.50kB) ⬇️
IntersectionType<'db>::from_two_elements_ 220.72kB 217.23kB -1.58% (3.49kB) ⬇️
is_redundant_with_impl 1.85MB 1.84MB -0.18% (3.33kB) ⬇️
infer_unpack_types 454.48kB 451.91kB -0.57% (2.57kB) ⬇️
IntersectionType 930.08kB 928.80kB -0.14% (1.27kB) ⬇️
Specialization 1.06MB 1.06MB -0.10% (1.09kB) ⬇️
... 16 more

prefect

Name Old New Diff Outcome
infer_expression_type_impl 14.45MB 14.17MB -1.93% (285.25kB) ⬇️
infer_expression_types_impl 59.94MB 59.67MB -0.45% (277.55kB) ⬇️
infer_definition_types 87.54MB 87.38MB -0.18% (164.13kB) ⬇️
all_negative_narrowing_constraints_for_expression 2.67MB 2.57MB -3.86% (105.53kB) ⬇️
all_narrowing_constraints_for_expression 6.94MB 6.85MB -1.32% (94.14kB) ⬇️
get_loop_header 536.12kB 476.76kB -11.07% (59.37kB) ⬇️
loop_header_reachability 482.88kB 445.89kB -7.66% (36.98kB) ⬇️
semantic_index 168.37MB 168.35MB -0.01% (23.16kB) ⬇️
is_redundant_with_impl::interned_arguments 5.58MB 5.58MB -0.12% (6.88kB) ⬇️
infer_scope_types_impl 51.65MB 51.64MB -0.01% (5.36kB) ⬇️
UnionType 3.61MB 3.60MB -0.13% (4.95kB) ⬇️
is_redundant_with_impl 5.61MB 5.60MB -0.07% (3.89kB) ⬇️
infer_unpack_types 872.58kB 869.27kB -0.38% (3.31kB) ⬇️
Type<'db>::member_lookup_with_policy_::interned_arguments 5.41MB 5.41MB -0.02% (936.00B) ⬇️
Type<'db>::member_lookup_with_policy_ 15.19MB 15.19MB -0.01% (912.00B) ⬇️
... 9 more

@astral-sh-bot
Copy link

astral-sh-bot bot commented Feb 24, 2026

ecosystem-analyzer results

Lint rule Added Removed Changed
invalid-assignment 0 1 0
Total 0 1 0

Full report with detailed diff (timing results)

@carljm carljm closed this Feb 24, 2026
@carljm carljm reopened this Feb 24, 2026
@oconnor663 oconnor663 force-pushed the jack/loop_header_bindings_in_loop_only branch from eab1bb6 to 495557c Compare February 24, 2026 20:22
@oconnor663
Copy link
Contributor Author

Rebased and applied the cleanup described here.

@AlexWaygood AlexWaygood removed their request for review February 24, 2026 20:23
@sharkdp
Copy link
Contributor

sharkdp commented Feb 25, 2026

I think the two ecosystem changes are worth looking into.

It took me a while to distill this MRE out of alerta, so I'm posting it here in case it's useful. It produces one additional diagnostic on main that is not present on your branch:

class Alert: ...

def f(*args, **kwargs):
    pass

def some_condition(*args, **kwargs) -> bool:
    return False

def process_action(alert: Alert, action: str):
    updated = None
    while some_condition(alert):
        if some_condition(1):
            updated = f(alert, action)
        elif some_condition(2):
            updated = f(alert, action)

        if isinstance(updated, Alert):
            updated = updated, action, 1

        if isinstance(updated, tuple):
            if len(updated) == 3:
                alert, action, timeout = updated
            elif len(updated) == 2:
                alert, action = updated

It would be good to understand if that diagnostic should go away.

And if that example is not strange enough, the diagnostic change in manticore is even weirder:

- manticore/core/smtlib/visitors.py:641:58: error[unsupported-operator] Operator `-` is not supported between objects of type `None | Unknown` and `None | Unknown`
+ manticore/core/smtlib/visitors.py:641:58: error[unsupported-operator] Operator `-` is not supported between two objects of type `None | Unknown`

Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you — the change sounds very plausible and the implementation looks good. I can certainly imagine that this solves some bugs as well. It might therefore be interesting to understand the ecosystem changes better. Maybe there is a useful regression test that we could add?

If those cases seem too exotic though, I'm definitely in favor of merging this, even without a deep analysis. This definitely seems more correct.

@carljm carljm removed their request for review February 25, 2026 20:07
@sharkdp
Copy link
Contributor

sharkdp commented Feb 27, 2026

I'm out for a few days, but the review here is finished, so I don't think we need to assign someone new(?). If there are significant changes, please unassign me and close/open if you feel it's necessary to review those.

@oconnor663
Copy link
Contributor Author

Will do, thanks David.

@oconnor663 oconnor663 force-pushed the jack/loop_header_bindings_in_loop_only branch from 495557c to 2a7d651 Compare March 3, 2026 06:29
@oconnor663
Copy link
Contributor Author

The manticore before/after diff seems to have gone away on rebase. Here's a further minimized version of the snippet @sharkdp posted above that's derived from the alerta ecosystem hit:

  def process(alert: int):
      while alert:
          if not alert:
              updated = 0
          if isinstance(updated, int):
              updated = updated, 1
          if isinstance(updated, tuple):
              if len(updated) == 2:
                  alert, _ = updated
              elif len(updated) == 1:
                  (alert,) = updated

With this change it's clean (the type of updated in the final branch is Never), but before this change it produces:

error[invalid-assignment]: Too many values to unpack
  --> x.py:11:17
   |
 9 |                 alert, _ = updated
10 |             elif len(updated) == 1:
11 |                 (alert,) = updated
   |                 ^^^^^^^^   ------- Got 2
   |                 |
   |                 Expected 1

This seems to be a really complicated cycle, probably something like the one described in this mdtest case where an instance of Divergent that shows up in a narrowing condition in an early iteration of the cycle leads to an un-narrowed type that participates in our "monotonic widening" strategy for cycle resolution. I haven't figured out all the details, but the high-level diff seems to be that simplifying the loopback bindings just makes the types in this cycle less complicated, and our normal filters for Divergent values are able to handle them.

@oconnor663 oconnor663 merged commit 0c88d7f into main Mar 3, 2026
51 checks passed
@oconnor663 oconnor663 deleted the jack/loop_header_bindings_in_loop_only branch March 3, 2026 08:02
carljm added a commit that referenced this pull request Mar 3, 2026
* main:
  [ty] Apply narrowing to walrus values (#23687)
  [`perflint`] Extend `PERF102` to comprehensions and generators (#23473)
  [ty] Fix GitHub-annotations mdtest output format (#23694)
  [ty] Reduce the number of potentially-flaky projects (#23698)
  [`pydocstyle`] Fix numpy section ordering (`D420`) (#23685)
  [ty] Move method-related types to a submodule (#23691)
  [ty] Avoid the mandatory "ecosystem-analyzer workflow run cancelled" notification every time you make a PR (#23695)
  [ty] Move `Type::subtyping_is_always_reflexive` to `types::relation` (#23692)
  Update conformance suite commit hash (#23693)
  [ty] Add mdtest suite for `typing.Concatenate` (#23554)
  [ty] filter out pre-loop bindings from loop headers (#23536)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ecosystem-analyzer ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants