We currently have wiki instructions for running Flake8 locally in a couple editors, including VS Code. This makes it convenient to correct Flake8 errors before pushing, but we currently don't have similar documentation for running mypy locally. Given how much effort we've been putting into adding mypy annotations to the codebase, we should make this easier to do.

Status quo

Running all mypy checks

Currently the canonical way to run mypy on PyTorch is via the test/test_type_hints.py file; for example:

$ python test/test_type_hints.py --save-xml=/tmp/mypy_time

On each of the dev machines I have access to, the above command (as of 5834438) takes about 4 minutes, the vast majority of which is for the test_run_mypy test (the one that uses mypy.ini, which applies to most of the Python files we typecheck):

$ test/print_test_stats.py /tmp/mypy_time | tail -n6
Total runtime is 0:04:07
4 longest tests of entire run:
    TestTypeHints.test_run_mypy  time: 165.97 seconds
    TestTypeHints.test_type_hint_examples  time: 40.09 seconds
    TestTypeHints.test_doc_examples  time: 29.65 seconds
    TestTypeHints.test_run_mypy_strict  time: 11.50 seconds

This is too long to include in a local development feedback loop. Note that even though mypy uses a cache by default, running our mypy test suite takes the same amount of time each run, because it runs mypy using 4 different configs (one for each test case) and thus invalidates .mypy_cache every time.

Running most mypy checks

If you're only editing files that use mypy.ini rather than mypy-strict.ini (which is most files in this repo), you can just run a single test from that test suite:

$ python test/test_type_hints.py TestTypeHints.test_run_mypy

After an initial warm-up run, this should only take a few seconds to complete. The runtime can go up significantly depending on what changes you've made since the last run; for instance, changing the return type of torch.random.set_rng_state from None to str increases the time to about 90 seconds:

diff --git a/torch/random.py b/torch/random.py
index 31e2643845..7d92aafb40 100644
--- a/torch/random.py
+++ b/torch/random.py
@@ -5,7 +5,7 @@ from torch._C import default_generator
 import torch
 
 
-def set_rng_state(new_state) -> None:
+def set_rng_state(new_state) -> str:
     r"""Sets the random number generator state.
 
     Args:

Running most mypy checks in VS Code

It is possible to enable mypy in VS Code by setting these values in .vscode/settings.json:

{
  "python.linting.enabled": true,
  "python.linting.mypyEnabled": true
}

This has the advantage of displaying the warnings inline after every edit rather than having to repeatedly re-run a command, and anecdotally seems to run a bit faster than running the command, although it can still be slow. To use the strict profile, add this setting as well:

  "python.linting.mypyArgs": [
    "--ignore-missing-imports",
    "--follow-imports=silent",
    "--show-column-numbers",
    "--config-file=/path/to/pytorch/mypy-strict.ini"
  ]

(The first three args are just the default VS Code mypy args.)

However, this setup has a couple problems (listed in decreasing order of importance):

• Regardless of the --config-file arg above, it shows mypy warnings even for files on which we disable mypy, such as test/test_testing.py. It seems that VS Code is ignoring the files entry in mypy.ini.
• It doesn't allow one to get correct warnings for both mypy.ini and mypy-strict.ini simultaneously (on different files). In fact, one seems to need to reload VS Code after switching which config file the --config-file arg in .vscode/settings.json points to (although I could be wrong about this).
• It has the potential to get out of sync with the tests in test/test_type_hints.py, since those do more than just run mypy with a given config file (e.g. they pass other args to mypy, just as in the VS Code config above).
• The --config-file argument is an absolute path, so the above VS Code config can't be simply copy-pasted between different machines (or checked into version control here).

Next steps?

It doesn't seem immediately obvious how to solve the first two issues without modifying the VS Code Python extension (or wherever the VS Code mypy integration is implemented). The third one may be a non-issue, though, and the fourth one is just a minor issue of convenience.

Should we include some of the above commands/configs in a wiki somewhere and leave this alone for now, or should we try to address some of these bullet points?

cc @ezyang @malfet @rgommers @xuzhao9 @gramster