Skip to content

Inclusion of tests in Python package installations is causing collisions #81482

@Shados

Description

@Shados

Describe the bug
Many Python packages store their tests in a top-level tests package, and include this in their sdist tarballs, which is typically what nixpkgs installs from. Combining said packages in a Python environment thus results in multiple tests packages being installed, with a decent chance of collision.

To Reproduce
Example collision:

shados@forcedperspective[~] λ nix-build -E 'with import <nixpkgs> { }; python38.withPackages(ps: with ps; [poetry])'
these derivations will be built:
  /nix/store/bfkjlxi7v6amg0kf2jyx69j3nkxnwkh4-python3-3.8.1-env.drv
building '/nix/store/bfkjlxi7v6amg0kf2jyx69j3nkxnwkh4-python3-3.8.1-env.drv'...
collision between `/nix/store/7fmcrha8ars9zq5lpakad3cf6yc2nmlv-python3.8-tomlkit-0.5.8/lib/python3.8/site-packages/tests/__pycache__/__init__.cpython-38.pyc' and `/nix/store/wmzbyb7r9qf4g3wqxamv1b228zdk603v-python3.8-cachy-0.3.0/lib/python3.8/site-packages/tests/__pycache__/__init__.cpython-38.pyc'
builder for '/nix/store/bfkjlxi7v6amg0kf2jyx69j3nkxnwkh4-python3-3.8.1-env.drv' failed with exit code 255
error: build of '/nix/store/bfkjlxi7v6amg0kf2jyx69j3nkxnwkh4-python3-3.8.1-env.drv' failed

Expected behavior
I would expect the above command to succeed, and produce a working Python environment.

Additional context
Arguably, Python packages that wish to include their tests in their sdist tarball should not be marking them as an installable package. However, this practice doesn't really cause any issues outside of Nix-land; tests are never run from the installed tests package, and we're unfortunately the only ones who care about file collisions.

As a result, I'm not sure we'd have much luck convincing package maintainers to alter their behaviour, so we might be better off working around the issue in nixpkgs. Possibly we could either prevent installation of tests somehow, or remove the tests package in post-install.

Metadata

  • system: "x86_64-linux"
  • host os: Linux 5.5.0, NixOS, 20.09.git.353f2215dc6 (Nightingale)
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.3.3
  • channels(shados): "home-manager"
  • channels(root): "nixos-20.09pre215333.42f0be81ae0"
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

@FRidh

Metadata

Metadata

Assignees

No one assigned

    Labels

    0.kind: bugSomething is broken6.topic: pythonPython is a high-level, general-purpose programming language.
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions