Tighten `dask-core` and `distributed` pinnings by charlesbluca · Pull Request #190 · conda-forge/dask-feedstock

charlesbluca · 2022-08-09T16:39:19Z

As part of addressing dask/dask#9367, this PR tightens the pinning of the dask-core and distributed dependencies so that we can only pull in packages with an exact version match, preventing us from erroneously pulling in nightly packages.

cc @galipremsagar

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-linter · 2022-08-09T16:39:25Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

charlesbluca · 2022-08-09T16:39:46Z

@conda-forge-admin, please rerender

github-actions · 2022-08-09T16:40:58Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/dask-feedstock/actions/runs/2826861231.

jakirkham · 2022-08-09T18:10:59Z

-    - dask-core {{ version }}
-    - distributed {{ version }}
+    - dask-core =={{ version }}
+    - distributed =={{ version }}


So one downside of this particular pinning is that we are not able to pick up a hot-fix from dask-core or distributed (for example 2022.8.1.1) without a corresponding release.

Yeah I was thinking about this - I wasn't sure how impactful this would be to the release process, considering that in the past, hotfixes have generally been done through a new patch release of all 3 packages. However, this obviously would limit our options moving forward.

Do you know of any way we can achieve the intended solution (excluding pre-alpha packages with the same version prefix) without using as tight of a pinning?

TIL dask-core {{ version }} and dask-core =={{ version }} are different -- thanks @charlesbluca!

Re: hot-fixes, I agree with @charlesbluca that I don't think this will be an issue in practice given our current release process. We've already just issued a new release of all three packages which would, I think, work with the existing changes here. The only other thing that comes to mind is security backports (Xref dask/community#244) which we've yet to ever act on. IIUC this could throw a wrench into the current proposal in that community issue, correct?

Toying around with conda install pinnings, it looks like we actually can do excludes by regex without creating a tight pinning; for example, doing something like:

conda install "dask=2022.7.1,!=2022.7.1a.*"

Would end up picking up all 2022.7.1.* packages excluding those that match 2022.7.1a.*. I assume that we won't be suffixing packages with a.* unless they're pre-releases (feel free to point out any cases where we might need this), so happy to try out this option here if it unblocks the security backport proposal 🙂

I assume that we won't be suffixing packages with a.* unless they're pre-releases

Yeah, that's my understanding too

jakirkham · 2022-08-09T18:12:35Z

cc @jrbourbeau (in case you have thoughts here :)

jakirkham

Generally LGTM. Had one suggestion below. Thanks Charles! 🙏

Co-authored-by: jakirkham <jakirkham@gmail.com>

jakirkham

Thanks Charles! 🙏

jakirkham · 2022-08-11T16:45:02Z

@jrbourbeau, could you please take another look? 🙂

jakirkham · 2022-08-12T17:41:03Z

Planning on merging EOD if no comments

jrbourbeau · 2022-08-12T17:45:12Z

Thanks @jakirkham for the ping -- I'm currently saturated with a few other things. Will plan to take a look at this in a bit, but also feel free to merge if that doesn't happen

jrbourbeau

Thanks @charlesbluca @jakirkham!

jakirkham · 2022-08-12T21:29:46Z

Thanks James! 🙏

@charlesbluca, it's worth noting that we will need a repodata hotfix to apply this to older package versions as well

charlesbluca · 2022-08-15T19:09:42Z

Might want to hold off on the repodata hotfix - I tried generating dask-core packages with the proposed security backport version scheme (2022.8.0.1 in this case) to see if they would work with the new conda-forge builds and it looks like that raises conflicts because we're pinning tightly to 2022.8.0:

$ mamba create -n test --override-channels -c ./main/ -c conda-forge dask==2022.8.0=*_1 dask-core==2022.8.0.1 --dry-run

Encountered problems while solving:
  - package dask-2022.8.0-pyhd8ed1ab_1 requires dask-core 2022.8.0,!=2022.8.0a*.*, but none of the providers can be installed

Need to play around with the conda recipe syntax a bit locally, but think we'll want something more like

    - dask-core {{ version }}.*,!={{ version }}a
    - distributed {{ version }}.*,!={{ version }}a

Not sure if we need to append a .* to these specifications or if that gets done manually; will try both out locally and open a new PR in each feedstock once I know the correct syntax.

EDIT:

Once that's done, we'll also want to make sure the repodata patch undoes the tight pinning we've set on these recent builds.

jakirkham · 2022-08-15T19:14:40Z

+    - dask-core ={{ version }},!={{ version }}a*
+    - distributed ={{ version }},!={{ version }}a*


This may need more tinkering, but this is one idea

Suggested change

- dask-core ={{ version }},!={{ version }}a*

- distributed ={{ version }},!={{ version }}a*

- dask-core =={{ version }}|{{ version }}.*,!={{ version }}a*

- distributed =={{ version }}|{{ version }}.*,!={{ version }}a*

Tighten dask-core and distributed pinnings

ffeaa35

charlesbluca requested review from TomAugspurger, alimanfoo, jakirkham, jcrist, jrbourbeau, koverholt, martindurant, mrocklin and pitrou as code owners August 9, 2022 16:39

galipremsagar approved these changes Aug 9, 2022

View reviewed changes

jakirkham reviewed Aug 9, 2022

View reviewed changes

jakirkham mentioned this pull request Aug 9, 2022

Errors being caused by allowing conda-forge & dask/label/dev packages to co-exist dask/dask#9367

Closed

Try using excludes regex to exclude pre-release packages

f7d81b0

jakirkham reviewed Aug 10, 2022

View reviewed changes

Comment thread recipe/meta.yaml Outdated

Correct exclude wildcard

7a79653

Co-authored-by: jakirkham <jakirkham@gmail.com>

jakirkham approved these changes Aug 11, 2022

View reviewed changes

jrbourbeau approved these changes Aug 12, 2022

View reviewed changes

jrbourbeau merged commit 9c2d84e into conda-forge:main Aug 12, 2022

jakirkham reviewed Aug 15, 2022

View reviewed changes

charlesbluca mentioned this pull request Aug 16, 2022

Fix dask-core / distributed pinning to properly exclude nightlies #191

Merged

5 tasks

charlesbluca mentioned this pull request Aug 16, 2022

Fix dask-core pinning to properly exclude nightlies conda-forge/distributed-feedstock#218

Merged

5 tasks

		- dask-core ={{ version }},!={{ version }}a*
		- distributed ={{ version }},!={{ version }}a*

Uh oh!

Conversation

charlesbluca commented Aug 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

conda-forge-linter commented Aug 9, 2022

Uh oh!

charlesbluca commented Aug 9, 2022

Uh oh!

github-actions Bot commented Aug 9, 2022

Uh oh!

jakirkham Aug 9, 2022

Choose a reason for hiding this comment

Uh oh!

charlesbluca Aug 9, 2022

Choose a reason for hiding this comment

Uh oh!

jrbourbeau Aug 9, 2022

Choose a reason for hiding this comment

Uh oh!

charlesbluca Aug 10, 2022

Choose a reason for hiding this comment

Uh oh!

jrbourbeau Aug 10, 2022

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Aug 9, 2022

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Aug 11, 2022

Uh oh!

jakirkham commented Aug 12, 2022

Uh oh!

jrbourbeau commented Aug 12, 2022

Uh oh!

jrbourbeau left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Aug 12, 2022

Uh oh!

charlesbluca commented Aug 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham Aug 15, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

charlesbluca commented Aug 9, 2022 •

edited

Loading

charlesbluca commented Aug 15, 2022 •

edited

Loading