Describe the bug
For a while now, some python3Packages.psutil routines have been segfaulting on darwin. This has become more problematic recently because newer versions of python3Packages.distributed make a call to psutil.disk_io_counters(), which pretty robustly segfaults on darwin. Packages that use python3Packages.distributed in their tests are failing as a result. e.g. https://hydra.nixos.org/log/x8a5k4lvfmnlyp088slxg0c72j1a7hq9-python3.8-stumpy-1.9.2.drv. python3Packages.psutil's own tests are of course disabled because they have proven to be so problematic.
We've opened some upstream issues for these problems (giampaolo/psutil#2009, giampaolo/psutil#1715), but I'm increasingly starting to feel these are Our Bugs.
The next step is probably to do some proper debugging on these segfaults, but I personally lack the motivation to learn darwin's weird debugging ecosystem (give me gdb and ELF anyday), so instead I'm documenting my findings here.
Steps To Reproduce
Steps to reproduce the behavior:
- Be on darwin (tested x86_64, macos 10.15)
$ nix-shell -p python38Packages.psutil --pure --run python
Python 3.8.8 (default, Jun 15 2021, 08:25:07)
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.disk_io_counters()
/private/tmp/nix-shell-4915-0/rc: line 1: 4918 Segmentation fault: 11 python
When I use a pip-installed version on the same machine it succeeds:
$ nix-shell -p python38 --pure
[nix-shell:~]$ python -m venv venv
[nix-shell:~]$ . venv/bin/activate
(venv)
[nix-shell:~]$ pip install psutil
Collecting psutil
Downloading psutil-5.8.0-cp38-cp38-macosx_10_9_x86_64.whl (236 kB)
|████████████████████████████████| 236 kB 23.5 MB/s
Installing collected packages: psutil
Successfully installed psutil-5.8.0
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/bob/venv/bin/python -m pip install --upgrade pip' command.
(venv)
[nix-shell:~]$ python
Python 3.8.8 (default, Jun 15 2021, 08:25:07)
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.disk_io_counters()
sdiskio(read_count=27110, write_count=309745, read_bytes=3164171264, write_bytes=6676201472, read_time=23219, write_time=58976)
>>>
Notify maintainers
@jonringer @NixOS/darwin-maintainers @teh @costrouc
Metadata
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute: python3Packages.psutil python3Packages.distributed python3Packages.stumpy
# a list of nixos modules affected by the problem
module:
Describe the bug
For a while now, some
python3Packages.psutilroutines have been segfaulting on darwin. This has become more problematic recently because newer versions ofpython3Packages.distributedmake a call topsutil.disk_io_counters(), which pretty robustly segfaults on darwin. Packages that usepython3Packages.distributedin their tests are failing as a result. e.g. https://hydra.nixos.org/log/x8a5k4lvfmnlyp088slxg0c72j1a7hq9-python3.8-stumpy-1.9.2.drv.python3Packages.psutil's own tests are of course disabled because they have proven to be so problematic.We've opened some upstream issues for these problems (giampaolo/psutil#2009, giampaolo/psutil#1715), but I'm increasingly starting to feel these are Our Bugs.
The next step is probably to do some proper debugging on these segfaults, but I personally lack the motivation to learn darwin's weird debugging ecosystem (give me gdb and ELF anyday), so instead I'm documenting my findings here.
Steps To Reproduce
Steps to reproduce the behavior:
When I use a pip-installed version on the same machine it succeeds:
Notify maintainers
@jonringer @NixOS/darwin-maintainers @teh @costrouc
Metadata
Maintainer information: