Skip to content

Better CPU affinity detection#12221

Merged
crusaderky merged 5 commits intodask:mainfrom
crusaderky:CPU_COUNT
Jan 28, 2026
Merged

Better CPU affinity detection#12221
crusaderky merged 5 commits intodask:mainfrom
crusaderky:CPU_COUNT

Conversation

@crusaderky
Copy link
Copy Markdown
Collaborator

@crusaderky crusaderky commented Jan 6, 2026

  • CPU_COUNT enhanced to reflect cpu affinity on Linux (all Python versions) and Windows/Mac (Python >=3.13) even when psutil is not installed
  • CPU_COUNT static type changed from int | None to int
  • Added unit tests for CPU affinity

This was tested on an enhanced CI matrix (71ff2c2) and returned all green (https://github.com/crusaderky/dask/actions/runs/20781477335/job/59679809102). CI changes were later reverted.

@crusaderky crusaderky force-pushed the CPU_COUNT branch 3 times, most recently from e3e1330 to 83ba052 Compare January 6, 2026 18:15
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 6, 2026

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 59dab32. ± Comparison against base commit 0a07553.

♻️ This comment has been updated with latest results.

@crusaderky crusaderky force-pushed the CPU_COUNT branch 3 times, most recently from 23137f9 to 8c314d3 Compare January 7, 2026 12:15
# Check cgroups if available
if sys.platform == "linux":
if LINUX:
quota, period = _try_extract_cgroup_cpu_quota()
Copy link
Copy Markdown
Collaborator Author

@crusaderky crusaderky Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope: this should belong to os.process_cpu_count. As it is complicated to test without mocking, I did not investigate if any of it is actually implemented. Let's revisit this after Python 3.13 becomes the minimum version in a few years.


from dask.system import cpu_count

psutil = pytest.importorskip("psutil")
Copy link
Copy Markdown
Collaborator Author

@crusaderky crusaderky Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run most tests when psutil is not installed

This reverts commit 71ff2c2.
@crusaderky crusaderky marked this pull request as ready for review January 7, 2026 12:39
@crusaderky
Copy link
Copy Markdown
Collaborator Author

@jacobtomlinson @jsignell does either of you have bandwidth to offer review?

Copy link
Copy Markdown
Collaborator

@dcherian dcherian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine from a quick look

@crusaderky crusaderky merged commit ff0ca1a into dask:main Jan 28, 2026
27 checks passed
@crusaderky crusaderky deleted the CPU_COUNT branch January 28, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants