Widen MutableMapping.update type by JelleZijlstra · Pull Request #6653 · python/typeshed

JelleZijlstra · 2021-12-22T05:56:31Z

Part of #6056

github-actions · 2021-12-22T06:05:37Z

Diff from mypy_primer, showing the effect of this PR on open source code:

bidict (https://github.com/jab/bidict)
+ bidict/_mut.py:150: error: Signature of "update" incompatible with supertype "MutableMapping"  [override]
+ bidict/_mut.py:150: note:      Superclass:
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, SupportsKeysAndGetItem[KT, VT], **kwargs: VT) -> None
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, Iterable[Tuple[KT, VT]], **kwargs: VT) -> None
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, **kwargs: VT) -> None
+ bidict/_mut.py:150: note:      Subclass:
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, Mapping[KT, VT], **kw: VT) -> None
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, Iterable[Tuple[KT, VT]], **kw: VT) -> None
+ bidict/_mut.py:150: note:          @overload
+ bidict/_mut.py:150: note:          def update(self, **kw: VT) -> None

websockets (https://github.com/aaugustin/websockets)
+ src/websockets/datastructures.py:137: error: Signature of "update" incompatible with supertype "MutableMapping"
+ src/websockets/datastructures.py:137: note:      Superclass:
+ src/websockets/datastructures.py:137: note:          @overload
+ src/websockets/datastructures.py:137: note:          def update(self, SupportsKeysAndGetItem[str, str], **kwargs: str) -> None
+ src/websockets/datastructures.py:137: note:          @overload
+ src/websockets/datastructures.py:137: note:          def update(self, Iterable[Tuple[str, str]], **kwargs: str) -> None
+ src/websockets/datastructures.py:137: note:          @overload
+ src/websockets/datastructures.py:137: note:          def update(self, **kwargs: str) -> None
+ src/websockets/datastructures.py:137: note:      Subclass:
+ src/websockets/datastructures.py:137: note:          def update(self, *args: Union[Headers, Mapping[str, str], Iterable[Tuple[str, str]]], **kwargs: str) -> None

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures.pyi:99: error: Signature of "update" incompatible with supertype "MutableMapping"
+ src/werkzeug/datastructures.pyi:99: note:      Superclass:
+ src/werkzeug/datastructures.pyi:99: note:          @overload
+ src/werkzeug/datastructures.pyi:99: note:          def update(self, SupportsKeysAndGetItem[K, V], **kwargs: V) -> None
+ src/werkzeug/datastructures.pyi:99: note:          @overload
+ src/werkzeug/datastructures.pyi:99: note:          def update(self, Iterable[Tuple[K, V]], **kwargs: V) -> None
+ src/werkzeug/datastructures.pyi:99: note:          @overload
+ src/werkzeug/datastructures.pyi:99: note:          def update(self, **kwargs: V) -> None
+ src/werkzeug/datastructures.pyi:99: note:      Subclass:
+ src/werkzeug/datastructures.pyi:99: note:          def update(self, *args: Union[Mapping[K, V], Iterable[Tuple[K, V]]], **kwargs: V) -> None
+ src/werkzeug/datastructures.pyi:277: error: Signature of "update" incompatible with supertype "MutableMapping"
+ src/werkzeug/datastructures.pyi:277: note:      Superclass:
+ src/werkzeug/datastructures.pyi:277: note:          @overload
+ src/werkzeug/datastructures.pyi:277: note:          def update(self, SupportsKeysAndGetItem[str, str], **kwargs: str) -> None
+ src/werkzeug/datastructures.pyi:277: note:          @overload
+ src/werkzeug/datastructures.pyi:277: note:          def update(self, Iterable[Tuple[str, str]], **kwargs: str) -> None
+ src/werkzeug/datastructures.pyi:277: note:          @overload
+ src/werkzeug/datastructures.pyi:277: note:          def update(self, **kwargs: str) -> None
+ src/werkzeug/datastructures.pyi:277: note:      Subclass:
+ src/werkzeug/datastructures.pyi:277: note:          def update(self, *args: Union[Mapping[str, Union[str, int]], Iterable[Tuple[str, Union[str, int]]]], **kwargs: Union[Union[str, int], Iterable[Union[str, int]]]) -> None

rclip (https://github.com/yurijmikhalevich/rclip)
- rclip/db.py:70: error: Argument 1 to "update" of "MutableMapping" has incompatible type "NewImage"; expected "Mapping[Any, None]"
+ rclip/db.py:70: error: Argument 1 to "update" of "MutableMapping" has incompatible type "NewImage"; expected "SupportsKeysAndGetItem[Any, None]"
+ rclip/db.py:70: note: Following member(s) of "NewImage" have conflicts:
+ rclip/db.py:70: note:     Expected:
+ rclip/db.py:70: note:         def __getitem__(self, Any) -> None
+ rclip/db.py:70: note:     Got:
+ rclip/db.py:70: note:         def __getitem__(self, str) -> object

ibis (https://github.com/ibis-project/ibis)
- ibis/backends/clickhouse/registry.py:720: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Type[ValueOp], Any]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/clickhouse/registry.py:720: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Type[ValueOp], Any]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"
- ibis/backends/clickhouse/registry.py:721: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Callable[[Any, Any, VarArg(Any)], Any]]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/clickhouse/registry.py:721: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Callable[[Any, Any, VarArg(Any)], Any]]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"
- ibis/backends/clickhouse/registry.py:722: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Type[UnaryOp], Callable[[Any, Any], Any]]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/clickhouse/registry.py:722: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Type[UnaryOp], Callable[[Any, Any], Any]]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"
- ibis/backends/clickhouse/registry.py:723: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/clickhouse/registry.py:723: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"
- ibis/backends/base/sql/registry/main.py:361: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/base/sql/registry/main.py:361: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"
- ibis/backends/impala/compiler.py:25: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "Mapping[Type[Node], Any]"
+ ibis/backends/impala/compiler.py:25: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[AnnotableMeta, Any]"; expected "SupportsKeysAndGetItem[Type[Node], Any]"

aioredis (https://github.com/aio-libs/aioredis)
- aioredis/client.py:4139: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Union[bytes, str], Optional[Any]]"; expected "Mapping[Union[bytes, str, memoryview], Callable[[Dict[str, str]], None]]"  [arg-type]
+ aioredis/client.py:4139: error: Argument 1 to "update" of "MutableMapping" has incompatible type "Dict[Union[bytes, str], Optional[Any]]"; expected "SupportsKeysAndGetItem[Union[bytes, str, memoryview], Callable[[Dict[str, str]], None]]"  [arg-type]

srittau

Primer shows that some third-party sub-classes need to be updated. That is ok.

aaugustin · 2022-03-20T21:34:04Z

Third-party maintainer (of websockets) ends ups here after half an hour spent tracking down what changed and is puzzled about how to fix this :-) What would you recommend?

Here's a trimmed down version of the code that is now raising a warning:

from typing import Iterable, Mapping, MutableMapping, Tuple, Union

class Headers(MutableMapping[str, str]):
    def update(
        self,
        *args: Union[Headers, Mapping[str, str], Iterable[Tuple[str, str]]],
        **kwargs: str,
    ) -> None:
        ...

I'd like to avoid introducing a dependency on typeshed or if typing.TYPE_CHECKING: ... conditionals — I don't have any at this point.

aaugustin · 2022-03-20T21:36:22Z

(If there's no simple fix, this will get a # type: ignore, given the dubious effort / benefit ratio of doing anything else.)

Refs python/typeshed#6653.

JelleZijlstra · 2022-03-20T21:50:52Z

So the problem with your version is that MutableMapping.update accepts any object that provides the keys and __getitem__ methods, but your override only accepts Mapping objects. You can see that in the default implementation of update(): https://github.com/python/cpython/blob/9d1c4d69dbc800ac344565119337fcf490cdc800/Lib/_collections_abc.py#L941

The most prominent example of an object that has keys and __getitem__ but isn't a Mapping is the pandas.Series class.

Possible fixes:

Use _typeshed.SupportsKeysAndGetItem, but you'll have to import it within an if TYPE_CHECKING block because it's stub-only.
Define your own version of the protocol in your code. It's pretty simple:

typeshed/stdlib/_typeshed/__init__.pyi

Line 77 in a11a664

class SupportsKeysAndGetItem(Protocol[_KT, _VT_co]):
Don't override update() at all, and rely on the stdlib's MutableMapping.update implementation.

aaugustin · 2022-03-21T08:12:13Z

Thank you very much!

I will investigate option 2; I didn't notice the type was that simple. (And double check that the implementation indeed supports the wider type; I think it does.)

Restore compatibility with the latest mypy. Refs python/typeshed#6653.

aaugustin · 2022-03-21T21:36:44Z

Fixed. Thanks again! :-)

Restore compatibility with the latest mypy. Refs python/typeshed#6653.

@overload

Mypy now includes python/typeshed#6653 which triggered ``` stubs/sortedcontainers/sorteddict.pyi:88: error: Signature of "update" incompatible with supertype "MutableMapping" [override] stubs/sortedcontainers/sorteddict.pyi:88: note: Superclass: stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, SupportsKeysAndGetItem[_KT, _VT], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: Subclass: stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Mapping[_KT, _VT], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, **kwargs: _VT) -> None ```

@overload

Mypy now includes python/typeshed#6653 which triggered ``` stubs/sortedcontainers/sorteddict.pyi:88: error: Signature of "update" incompatible with supertype "MutableMapping" [override] stubs/sortedcontainers/sorteddict.pyi:88: note: Superclass: stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, SupportsKeysAndGetItem[_KT, _VT], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: Subclass: stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Mapping[_KT, _VT], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None stubs/sortedcontainers/sorteddict.pyi:88: note: @overload stubs/sortedcontainers/sorteddict.pyi:88: note: def update(self, **kwargs: _VT) -> None ```

JelleZijlstra and others added 2 commits December 21, 2021 21:56

Widen MutableMapping.update type

26aa303

Part of #6056

[pre-commit.ci] auto fixes from pre-commit.com hooks

59311b5

JelleZijlstra mentioned this pull request Dec 22, 2021

dict.__init__() and dict.update() don't match #6056

Open

srittau approved these changes Dec 22, 2021

View reviewed changes

srittau merged commit 14268f6 into master Dec 22, 2021

srittau deleted the JelleZijlstra-patch-2 branch December 22, 2021 07:54

aaugustin added a commit to python-websockets/websockets that referenced this pull request Mar 20, 2022

Work around problematic change in typeshed.

db600e5

Refs python/typeshed#6653.

aaugustin added a commit to python-websockets/websockets that referenced this pull request Mar 21, 2022

Widen HeadersLike type.

7210324

Restore compatibility with the latest mypy. Refs python/typeshed#6653.

aaugustin added a commit to python-websockets/websockets that referenced this pull request Mar 23, 2022

Widen HeadersLike type.

747df3d

Restore compatibility with the latest mypy. Refs python/typeshed#6653.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Widen MutableMapping.update type#6653

Widen MutableMapping.update type#6653
srittau merged 2 commits intomasterfrom
JelleZijlstra-patch-2

JelleZijlstra commented Dec 22, 2021

Uh oh!

github-actions bot commented Dec 22, 2021

Uh oh!

srittau left a comment

Uh oh!

aaugustin commented Mar 20, 2022

Uh oh!

aaugustin commented Mar 20, 2022

Uh oh!

JelleZijlstra commented Mar 20, 2022

Uh oh!

aaugustin commented Mar 21, 2022

Uh oh!

aaugustin commented Mar 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

JelleZijlstra commented Dec 22, 2021

Uh oh!

github-actions bot commented Dec 22, 2021

Uh oh!

srittau left a comment

Choose a reason for hiding this comment

Uh oh!

aaugustin commented Mar 20, 2022

Uh oh!

aaugustin commented Mar 20, 2022

Uh oh!

JelleZijlstra commented Mar 20, 2022

Uh oh!

aaugustin commented Mar 21, 2022

Uh oh!

aaugustin commented Mar 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants