The python 3.7 tests fail because that version doesn't have typing.TypedDict. Type checking would still work correctly on all supported versions if instead typing_extensions.TypedDict is used, however I can't get that to work in CI. Any suggestions? :)

This seems like a bug in MyPy or TypedDict. It doesn't make sense that we'd add what should be a subclass of dict to the union which already has dict. Please report this to MyPy and/or Python first.

Seems like this is relevant in MyPy: python/mypy#12934

We use typing_extensions in other places. What does "can't get it to work" mean?

TypedDict, either from typing_extensions or with MyPy set to a higher Python version, does not appear to be a valid type on its own. So there is no way to merge this right now as it introduces errors (or ambiguity if ignored). It seems likely that python/mypy#12934 or another related issue is what needs to be fixed.

We use typing_extensions in other places. What does "can't get it to work" mean?

CI reports ModuleNotFoundError: No module named 'typing_extensions' on 3.8 and later: https://github.com/pallets/flask/runs/7337221340

This seems like a bug in MyPy or TypedDict. It doesn't make sense that we'd add what should be a subclass of dict to the union which already has dict. Please report this to MyPy and/or Python first.

Seems like this is relevant in MyPy: python/mypy#12934

I don't believe it's a bug in MyPy or TypedDict per se. According to its typing interface, a TypedDict exposes fewer methods than a dict does, even though at runtime it's really just a dict. So from a typing perspective it makes sense to not allow a TypedDict to be accepted by a function that accepts a Dict. Otherwise, this would be valid:

class DefinitelyContainsStatus(TypedDict):
    status: str

def delete_status(d: dict):
    del d['status']

d: DefinitelyContainsStatus = {'status': 'ok'}
delete_status(d)

Now you'd have a DefinitelyContainsStatus dict that doesn't contain status.

Flask doesn't do anything of the sort; it doesn't modify the dict at all. So it should be able to accept dicts typed as TypedDict.

However, you are right about the ambiguity. Perhaps the right approach is not to add TypedDict to ResponseValue, but instead to replace Dict[str, Any] with Mapping[str, Any]? That does accept TypedDicts.

No, we specifically accept dict, not anything that satisfies the mapping protocol.

See my subsequent comments. This is an issue with MyPy or TypedDict, there is nothing that can be done in Flask. Either it's an issue because TypedDict isn't valid where dict is, or it's an issue because the TypedDict base class on its own is not a valid type in unions.

In my testing typing._TypedDict is a valid type in unions, and MyPy does consider actual TypedDicts to be a member of that type and therefore a member of the union. So that would solve the problem for 3.8 and later.

However Python 3.7 still poses a problem, there's no TypedDict in the 3.7 standard library and I don't see a way to get it working with the one provided by typing_extensions.

Would it be agreeable to add a version check, so that on 3.7 the current union is used and for 3.8+ typing._TypedDict is added? Similar version-dependent code exists for example in src/flask/app.py for iscoroutinefunction.

typing_extensions imports fine:

if t.TYPE_CHECKING:
    import typing_extensions as te

...
    "te.TypedDict"

I've already tried all this, it makes other things fail, which also fail with typing.TypedDict. I don't plan to add a dependency on some _ prefixed type either, that seems likely to break. Take it up with MyPy.

That doesn't import fine, because the import is done within the TYPE_CHECKING guard but the union is defined outside. CI fails when done that way: https://github.com/pallets/flask/actions/runs/2669339095

Can you tell me more about the failures you're referring to? Aside from the Python 3.7 based environments nothing appears to break in CI. It doesn't seem to fail for our Flask application either which uses TypedDicts all over the place.

You need to put it in quotes if it's conditionally imported.

I'm not sure what CI you're looking at. The CI for this PR has failures everywhere. When I fix the TypedDict import, it causes more failures, which then cascade to other areas.

src/flask/typing.py:16: error: Variable "typing_extensions.TypedDict" is not valid as a type  [valid-type]
src/flask/typing.py:16: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

I'm looking at https://github.com/pallets/flask/actions/runs/2670388875 (which I believe is 370b0cb, it doesn't say on that page), which does not use typing_extensions.

I've added the version check in 261c002, but since the PR is closed CI doesn't run. However I believe all tests should pass, since they did before for 3.8+ and the version check fixes the 3.7 case.

Meanwhile, if you don't see a way forward for this particular PR, what other approaches would you consider acceptable for solving this regression?

Report the issue to MyPy (or lend your feedback to the issue I already linked). It's an issue with MyPy or typing in general, not with Flask.

It's not a issue from MyPy's point of view, as I've detailed above in #4695 (comment).

Being able to pass a TypedDict to an interface that accepts dict, means that this interface can modify the dict without regard to the type invariants afforded by TypedDict, so any remaining references to that TypedDict cannot rely on its type invariants still being held. Therefore, the MyPy error is correct and this is not an issue with the MyPy project.

Also, the MyPy project will tell you that Mapping is the appropriate type to accept, since Flask does not modify the dict and TypedDicts do satisfy the Mapping protocol.

Mapping is the appropriate type to accept, since Flask does not modify the dict

Mapping is not the appropriate type. Flask only has behavior for dict, not for any arbitrary class that satisfies the mapping protocol (which can be other things besides subclasses of dict).

Do you suppose that type checking is the right place to enforce this, at the cost of not being able to solve real-world problems? You can already return any non-dict Mapping if you simply don't use type checkers and it will still break visibly.

Right now, Flask's typing.py rejects valid real-world usage. Adding Mapping solves this real-world problem and makes the typing situation strictly better than it was in 2.1.2.

Do you suppose that type checking is the right place to enforce this

Yes, given that the entire point of type checking is to describe the correct types and catch things before runtime. If they're not able to do that, that's a design issue that needs to be fixed with typing or type checkers.

Right now, Flask's typing.py rejects valid real-world usage.

But with Mapping, it would accept invalid uses, so that doesn't seem "strictly better" just "different for that specific use case".

Yes, given that the entire point of type checking is to describe the correct types and catch things before runtime. If they're not able to do that, that's a design issue that needs to be fixed with typing or type checkers.

In that case, I would suggest to revert #4579.

But with Mapping, it would accept invalid uses, so that doesn't seem "strictly better" just "different for that specific use case".

But in 2.1.2, literally every invalid type was accepted. Yes, being able to reject all invalid types before runtime is ideal, but not at the cost of also rejecting valid types.

With Mapping, all invalid non-Mapping types are still rejected, and non-dict Mapping types cause a very clearly written runtime error: The view function did not return a valid response. The return type must be a string, dict, list, tuple with headers or status, Response instance, or WSGI callable, but it was a object. which cannot be misconstrued.

Surely that is still a significant improvement over 2.1.2.

In 2.1.3, where TypedDicts are not accepted, we would lose the ability to strongly type check our JSON-based API endpoints.

I've pushed a change to use typing_extensions.TypedDict. I've also temporarily bumped MyPy's Python version to 3.8. Notice that the error "TypedDict is not valid as a type" is shown whether you use it from typing or typing_extensions.

src/flask/typing.py:15: error: Variable "typing_extensions.TypedDict" is not valid as a type  [valid-type]

src/flask/typing.py:15: error: Variable "typing.TypedDict" is not valid as a type  [valid-type]

The fact that it fails with MyPy in 3.8 mode suggests that the sys.version check is just masking the type completely, hiding the new issue. So that doesn't seem like a valid solution.

If this cannot be solved without using Mapping, that needs to be reported to typing, as it's forcing a less precise type when that shouldn't be required. Please open an issue about that and link it here before I merge this.

Well, I definitely agree there's a contravariance problem when sending a TypedDict to a dict receiver, but I'm not sure what the typing or MyPy projects could theoretically do to resolve this.

Suppose they could define a WeakTypedDict that's still a dict at runtime and that can validly be sent to a dict receiver. But then type checkers would lose the ability to reason correctly about the contents of that dict, which isn't a desirable outcome.

Still, I shall open a issue there and link it here.

For Flask's purpose though, it doesn't matter what the contents are. You want to check the return value of your own function, where it does matter. But route and make_response only care that it's a dict, and specifically a dict and not another mapping.

They could also add an "immutable dict" to be able to say "any dict type, and I won't change the contents so it's valid for typed dict too". Basically mapping but specifically for dict.

python/mypy#13122

Thanks. I suppose mapping is ok since there's still an error at runtime, although hopefully typing can address it eventually. I'll clean this up in a bit.