Use declarative setuptools by graingert · Pull Request #7629 · dask/distributed

graingert · 2023-03-08T16:06:28Z

mostly built by running pipx run setup-py-upgrade . and pipx run versioneer install --no-vendor with some manual tweaks

related to #7622

Tests added / passed
Passes pre-commit run --all-files

jrbourbeau

Thanks for working on this @graingert. Do you have bandwidth to do the same things for dask/dask? dask/dask#10032

graingert · 2023-03-08T16:15:21Z

this fails with:

The reported errors are:
- Encountered problems while solving:
-   - nothing provides requested dask-core 2023.3.1a230308
- 

Leaving build/test directories:
  Work:
 /usr/share/miniconda3/envs/test/conda-bld/work 
  Test:
 /usr/share/miniconda3/envs/test/conda-bld/test_tmp 
Leaving build/test environments:
  Test:
source activate  /usr/share/miniconda3/envs/test/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p 
  Build:
source activate  /usr/share/miniconda3/envs/test/conda-bld/_build_env 


Traceback (most recent call last):
  File "/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/boa/cli/mambabuild.py", line 141, in mamba_get_install_actions
    solution = solver.solve_for_action(_specs, prefix)
  File "/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/boa/core/solver.py", line 230, in solve_for_action
    t = self.solve(specs)
  File "/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/boa/core/solver.py", line 220, in solve
    raise RuntimeError("Solver could not find solution." + error_string)
RuntimeError: Solver could not find solution.Mamba failed to solve:
 - dask-core 2023.3.1a230308
 - python >=3.8
 - pip

jrbourbeau · 2023-03-08T17:23:38Z

Hrm, there's usually some delay between packages showing up on the web and them being available for download (something to do with CDN refreshing). I'll restart the conda build CI job to see if it passes now

jrbourbeau · 2023-03-08T17:29:43Z

Okay, so it's still failing, but got further in the build process

EDIT: ModuleNotFoundError: No module named 'versioneer'

graingert · 2023-03-08T17:47:59Z

yeah it looks like pip is ignoring the [build-system] requires

graingert · 2023-03-08T18:29:49Z

it looks like boa disables pip build isolation, https://github.com/mamba-org/boa/blob/cd087c5582e11125fc0a8bc855056de5b4b41ddf/boa/core/build.py#L399-L406 which is needed for pyproject.toml build-system requires to work

jrbourbeau · 2023-03-08T18:49:40Z

cc @jakirkham @charlesbluca in case you have any thoughts on making pyproject.toml + boa work well together

github-actions · 2023-03-08T19:00:14Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

      26 files +      1       26 suites +1 12h 8m 39s ⏱️ - 20m 6s
  3 543 tests ±      0   3 437 ✔️ +      1   104 💤 ±  0 2 ❌ - 1
44 822 runs +1 320 42 726 ✔️ +1 246 2 094 💤 +75 2 ❌ - 1

For more details on these failures, see this check.

Results for commit 1280220. ± Comparison against base commit 0d39c19.

♻️ This comment has been updated with latest results.

.github/workflows/conda.yml

setup.cfg

jakirkham · 2023-03-08T19:34:19Z

cc @jakirkham @charlesbluca in case you have any thoughts on making pyproject.toml + boa work well together

There shouldn't be anything to do here

jrbourbeau · 2023-03-08T19:44:11Z

Yeah, sorry @graingert fixed that in d3106d0

graingert · 2023-03-09T12:21:19Z

looks like we're getting hit by python/cpython#99130 the recommendation is to use importlib_metadata on platforms with a buggy importlib.metadata.entry_points implementation what do you think @jrbourbeau ?

jakirkham · 2023-03-09T17:04:00Z

Not James, but that seems sensible to me

jrbourbeau · 2023-03-13T18:05:26Z

Not James

But even better!

@graingert FWIW using importlib_metadata in these cases seems reasonable to me too

jakirkham

Thanks Thomas! 🙏

Generally this looks good. Had a couple minor questions below

setup.cfg

.github/workflows/conda.yml

jakirkham

LGTM. Thanks Thomas! 🙏

Co-authored-by: jakirkham <jakirkham@gmail.com>

jakirkham

Thanks Thomas! 🙏

This is looking pretty good. Had one minor comment below

continuous_integration/recipes/distributed/meta.yaml

…ve-config

jrbourbeau

Thanks @graingert! Overall this looks great.

distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated and distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers seems to be failing consistently in CI here but also don't appear in the flaky test reports. I opened #7717 to double check that those failures aren't on main. Maybe some modification was made when copying over the pytest filterwarnings?

jrbourbeau · 2023-03-28T16:00:53Z

Yeah, it looks like those tests are passing over in #7717. So it appears they're related to the changes here. Again, my guess is it's related to our filterwarnings

graingert · 2023-03-28T16:21:14Z

@jrbourbeau nice catch, I diffed the value of pprint.pp(warnings.filters) before and after my changes and got:

   None,
   0),
  ('ignore',
-  re.compile('(?s)Exception in thread.*old_ssh.*channel\\.send\\(b"\\\\x03"\\).*Socket is closed', re.IGNORECASE|re.DOTALL),
+  re.compile('(?s)Exception in thread.*old_ssh.*channel\\.send\\(b"\\x03"\\).*Socket is closed', re.IGNORECASE|re.DOTALL),
   <class 'pytest.PytestUnhandledThreadExceptionWarning'>,
   None,
   0),

charlesbluca

Thanks @graingert 😄 a couple questions, mostly around conda build stuff:

.github/workflows/conda.yml

charlesbluca · 2023-03-28T17:27:26Z

continuous_integration/recipes/distributed/meta.yaml

    - pip
    - dask-core {{ dask_version }}
+    - versioneer =0.28
+    - tomli # [py<311]


Is the version selector necessary here? IIRC this is a noarch build

If we build on 3.11 tomli isn't needed, my understanding is noarch builds still build a package for each python version?

While true, versioneer has try...except... logic to prefer the builtin when available. So it probably doesn't matter much whether the selector is here or not

Think Charles' point is we generally don't want to use selectors in a noarch file since it can't really be conditioned. That said, with build & host dependencies this is less relevant. Since those are just used for the build and the packages wind up with a fixed set of dependencies regardless

This all to say there are valid reasons to drop this selector or keep it. However in terms of the end result, it shouldn't matter which way we go

Thanks for the clarification @jakirkham - I'm fine going either way here, just wanted to verify that this had no impact on the build

charlesbluca · 2023-03-28T17:52:58Z

pyproject.toml

+-p no:legacypath'''
+filterwarnings = [
+    "error",
+    '''ignore:Please use `dok_matrix` from the `scipy\.sparse` namespace, the `scipy\.sparse\.dok` namespace is deprecated.:DeprecationWarning''',


Nit for readability - is there a reason all of these are triple-single-quoted? Assuming some of these are for character escaping purposes (not too familiar with TOML syntax), but this example pyproject.toml seems to imply that a lot of these could be double quoted instead:

https://github.com/rapidsai/dask-cuda/blob/9fef6b7b1a65ab0a5bd5784c440451d8fcb2d71e/pyproject.toml#L121-L131

A few of these have literal backslash escapes and single quotes in and I wanted the indentation to line up across all of them

Ah thanks for the clarification, didn't realize that pytest requires literal backslashes in the filter strings. In that case understand that it's probably best to stick to one TOML string type and stay consistent

…ve-config

jakirkham · 2023-03-30T07:56:54Z

Any more thoughts on this one?

jrbourbeau

Thanks @graingert for all your work here -- and @jakirkham @charlesbluca for the reviews

jakirkham · 2023-03-30T16:49:01Z

Thanks James! 🙏

jrbourbeau reviewed Mar 8, 2023

View reviewed changes

graingert force-pushed the use-declarative-config branch from c1d9b86 to ee06f0e Compare March 8, 2023 16:38

graingert mentioned this pull request Mar 8, 2023

Move from setup.py to pyproject.toml #7622

Open

jrbourbeau reviewed Mar 8, 2023

View reviewed changes

.github/workflows/conda.yml Outdated Show resolved Hide resolved

jrbourbeau reviewed Mar 8, 2023

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

graingert force-pushed the use-declarative-config branch from 00057b2 to c2e9fc7 Compare March 9, 2023 11:31

graingert mentioned this pull request Mar 15, 2023

dask cli UserWarning when using entry points registered by pep660 editable installs from setuptools dask/dask#10068

Closed

graingert force-pushed the use-declarative-config branch 3 times, most recently from 8aae43e to e6372d5 Compare March 16, 2023 16:12

graingert marked this pull request as ready for review March 16, 2023 16:13

graingert requested a review from fjetter as a code owner March 16, 2023 16:13

graingert force-pushed the use-declarative-config branch from b6394dd to a49e49d Compare March 16, 2023 16:45

graingert marked this pull request as draft March 16, 2023 16:54

graingert marked this pull request as ready for review March 16, 2023 16:58

jakirkham reviewed Mar 16, 2023

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

setup.cfg Show resolved Hide resolved

.github/workflows/conda.yml Show resolved Hide resolved

graingert requested a review from jakirkham March 17, 2023 11:55

jakirkham approved these changes Mar 17, 2023

View reviewed changes

graingert and others added 5 commits March 22, 2023 18:02

fix VCS case in versioneer config

c1f1f26

Update .github/workflows/conda.yml

588ddc0

Co-authored-by: jakirkham <jakirkham@gmail.com>

remove duplicate codespell config from setup.cfg

f10b451

note flake8's lack of pyproject.toml support

6d9940f

move coveragerc into pyproject.toml

a57f074

graingert force-pushed the use-declarative-config branch from 9bfc011 to a57f074 Compare March 22, 2023 18:02

jakirkham reviewed Mar 27, 2023

View reviewed changes

continuous_integration/recipes/distributed/meta.yaml Show resolved Hide resolved

Merge branch 'main' of github.com:dask/distributed into use-declarati…

4860d0b

…ve-config

jrbourbeau reviewed Mar 28, 2023

View reviewed changes

fix test_old_ssh_nprocs_renamed_to_n_workers warning filter

65392c3

charlesbluca reviewed Mar 28, 2023

View reviewed changes

graingert added 3 commits March 29, 2023 14:38

move flake8 config to .flake8

df5329f

remove outdated references to setup.cfg

853b636

remove --channel conda-forge from mambabuild

50a268b

graingert requested a review from jrbourbeau March 29, 2023 16:02

graingert and others added 2 commits March 29, 2023 17:10

Merge branch 'main' of github.com:dask/distributed into use-declarati…

4cd6108

…ve-config

Rerun CI

1280220

jakirkham mentioned this pull request Mar 30, 2023

Upgrade versioneer to 0.28 #7214

Closed

2 tasks

jrbourbeau changed the title ~~use declarative setuptools~~ Use declarative setuptools Mar 30, 2023

jrbourbeau approved these changes Mar 30, 2023

View reviewed changes

jrbourbeau merged commit 78a926d into dask:main Mar 30, 2023

graingert deleted the use-declarative-config branch March 30, 2023 16:01

graingert mentioned this pull request Apr 5, 2023

Merge identical tool.mypy.overrides sections #7749

Merged

2 tasks

This was referenced Apr 14, 2023

Release 2023.4.0 conda-forge/distributed-feedstock#249

Merged

namespace packages take priority over setuptools PEP 660 editables in lenient mode #7782

Open

charlesbluca mentioned this pull request Nov 21, 2023

crick source install blocking 3.9 CI #8369

Closed

Uh oh!

Conversation

graingert commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrbourbeau left a comment

Choose a reason for hiding this comment

Uh oh!

graingert commented Mar 8, 2023

Uh oh!

jrbourbeau commented Mar 8, 2023

Uh oh!

jrbourbeau commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graingert commented Mar 8, 2023

Uh oh!

graingert commented Mar 8, 2023

Uh oh!

jrbourbeau commented Mar 8, 2023

Uh oh!

github-actions bot commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

Uh oh!

Uh oh!

jakirkham commented Mar 8, 2023

Uh oh!

jrbourbeau commented Mar 8, 2023

Uh oh!

graingert commented Mar 9, 2023

Uh oh!

jakirkham commented Mar 9, 2023

Uh oh!

jrbourbeau commented Mar 13, 2023

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jrbourbeau left a comment

Choose a reason for hiding this comment

Uh oh!

jrbourbeau commented Mar 28, 2023

Uh oh!

graingert commented Mar 28, 2023

Uh oh!

charlesbluca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

charlesbluca Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

graingert Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

jakirkham Mar 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charlesbluca Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

charlesbluca Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

graingert Mar 29, 2023

Choose a reason for hiding this comment

graingert commented Mar 8, 2023 •

edited

Loading

jrbourbeau commented Mar 8, 2023 •

edited

Loading

github-actions bot commented Mar 8, 2023 •

edited

Loading

jakirkham Mar 28, 2023 •

edited

Loading