Install python libraries with pip 20.2 and the 2020-resolver planned for the 20.3 release. See the changes to the pip dependency resolver in 20.2 (2020)

This fails for me when installing in the docker image, and when installing on macOS in a virtualenv. It succeeds when the default resolver in pip 20.2 is used.

The error with --use-feature=2020-resolver is:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r requirements/shared.txt (line 9))

However, idna==2.10, with hashes, is defined in requirements/constraints.txt.

I'm going to submit this to the pip developers as a potential use case for the 2020-resolver, so here's some background. There are two scenarios when we install requirements:

• For development and production, we install all the requirements. Some of these require binary development libraries, such as the MySQL client libraries.
• To build documentation on https://ichnaea.readthedocs.io, we just need a subset of the requirements, and none of the binary development libraries.

To handle this, we have 4 requirements files:

• requirements/default.txt - The top-level requirements for production and development
• requirements/docs.txt -The top-level documentation requirements (Sphinx and a Sphinx theme)
• requirements/shared.txt - Top-level requirements used by both production and docs (certifi for certs, everett for configuration, requests[security], and urllib3[secure])
• requirements/constraints.txt - Second-level requirements without [extra] specifiers

We specify hashes in all four files, using hashin to populate them.

Our requirements files include other files:

• default.txt: -c constraints.txt, -r docs.txt, -r shared.txt
• docs.txt: -c constraints.txt, -r shared.txt
• shared.txt: None
• contraints.txt: None

This allows us to define each requirement exactly once, and ensure that docs are built with the same versions in the development environment and CI as on ReadTheDocs.

This allows us to install requirements with pip install -r requirements/default.txt for development or for creating Docker images, and pip install -r requirements/docs.txt on ReadTheDocs. It works with the default resolver through pip 20.2. With pip --use-feaature=2020-resolver, it seems that the hash checker gets confused about some constraints.

It seems to be related to constraints that are required in two paths from two different files (determined with pipdeptree). For example, idna is included twice:

• Sphinx (docs.txt)
  • requires requests (shared.txt)
    • requires idna (constraints.txt)
• geoip2 (default.txt)
  • requires aiohttp (constraints.txt)
    • requires yarl (constraints.txt)
      • requires idna (constraints.txt)

I've also seen chardet have the --require-hashes mode when installing in a virtualenv (system packages allowed) on macOS. It has similar multiple requires paths:

• Sphinx (docs.txt)
  • requires requests (shared.txt)
    • requires chardet (constraints.txt)
• geoip2 (default.txt)
  • requires aiohttp (constraints.txt)
    • requires chardet (constraints.txt)