MAINT: backports/prep for 1.13.1 by tylerjereddy · Pull Request #20632 · scipy/scipy

tylerjereddy · 2024-05-02T22:27:37Z

We're about 3.5 weeks from the planned branching for SciPy 1.14.x. Since 1.13.0 was a bit rushed to support NumPy 2.x, and indeed I made a few oversights for backporting, it may make sense to at least assess how much work it would be to deliver 1.13.1. I've cleaned up the backport labels a bit too, since there were almost 50 of them. Many were things I could have backported to 1.12.x, but I think that would stretch me (and reviewers/support) too thin at this point.

Backports included here (so far):

Backports already merged to the release branch:

TODO:

* Fixes scipygh-20300 by syncing `pocketfft` again, this time to completely disable `aligned_alloc`; see scipy/pocketfft#3 for details, but in short our more conservative shim was not sufficient for `conda-forge`, so let's just do the same thing NumPy did... [skip cirrus] [skip circle]

…`. (scipy#20401)

[doc only]

This was reported to cause test failures under windows + MKL in conda-forge cf scipy#20471

A slight tolerance violation was reported on conda-forge in scipy#20472

A small tolerance violation reported on conda-forge in scipy#20472

An exact equality was reported as flaky on conda-forge in scipy#20472 The tolerance violations are of the order of fp noise (< 2e-16), and I don't think that pickling/unpickling guarantees bit-to-bit compatibility. In principle, this may invoke recalculations and those may be subject to fp noise. So I think it's OK to only require allclose(atol=eps) instead of exact equality.

tol violation observed on conda-forge on win+blis; suggested in scipy#20474 (comment)

f2py check was just too strict; LAPACK docs indicate that for N=1, lwork>=1 is acceptable: * LWORK (input) INTEGER * The dimension of the array WORK. * If N <= 1, LWORK must be at least 1. * If JOBZ = 'N' and N > 1, LWORK must be at least 2*N+1. * If JOBZ = 'V' and N > 1, LWORK must be at least * 1 + 6*N + 2*N**2. https://www.netlib.org/lapack/explore-3.1.1-html/ssyevd.f.html closes scipygh-20512

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>

…ith an infinite weight (scipy#20573) * Fix exact rotations at 180 deg, improve near 180 deg Comments * Tests for exact near 180 deg rotations * Fix tests * Code review updates --------- Co-authored-by: Scott Shambaugh <scottshambaugh@users.noreply.github.com>

* Apply "old version" of the patch provided at scipygh-20527

* Update the SciPy `1.13.1` release notes following a series of backports. [skip cirrus]

Showed up as a linting error in an unrelated PR for me: ``` scipy/interpolate/_interpolate.py:918:30: UP032 [*] Use f-string instead of `format` call scipy/interpolate/_interpolate.py:1972:30: UP032 [*] Use f-string instead of `format` call ``` This should not happen; the old code is fine, so this check needs to be silenced or fixed separately. Similar to scipygh-20601. [skip ci]

* Fixes scipygh-19423 * Add a few more `case` statements to account for the (i.e., Windows) data types that don't have a fixed width, and add a regression test. [skip circle] [skip cirrus]

* attempt to deal with scipygh-20576

* some minor import fixes following a large series of backports

…0643)" This reverts commit ee6203d.

* more import cleanups after backport activity to make the linter happy

tylerjereddy · 2024-05-15T21:02:40Z

I've added substantially more backports and a few manual shims--we're back at parity with merged backport-labeled PRs.

Let's see where the regular CI lands. I did have to revert and unlabel one PR (gh-20643) based on local testing with clang-16 on Mac, but the rest made it in, so far. I expanded the TODO list above based on known CI failure in main, etc.

Edit: between gh-20727 and gh-20533, I suspect we'll be able to get the regular CI passing today.

[skip circle]

* We're seeing CI failures related to an undesirable bump to Python `3.12` in this job, when the intention was clearly to respect the Python version specific in the GHA matrix. I didn't check too closely why exactly it suddenly started happening, but some packages weren't ready for `3.12` yet on this job (`scikit-umfpack` in particular) and I don't see too much harm in adding an extra pin to respect the intention for the Python version.

* attempt to fix `M1 test - openblas` MacOS ARM CI job based on some `gfortran` shims applied to a similar job on `main`

tylerjereddy · 2024-05-19T01:20:58Z

The regular CI is finally passing here (1 or 2 known flakes sometimes require restarts).

I'm going to proceed with a test pass of the wheel builds before updating the release notes again, because we may see shenanigans with the bump to OpenBLAS 0.3.27 for example.

* This is an empty commit to test the wheel building in the above PR. [wheel build]

tylerjereddy · 2024-05-19T01:49:29Z

For MacOS ARM wheel failure line 13: gfortran: command not found, I may need similar shims to the regular CI Mac ARM job changes I made (based on a subset of changes from @thalassemia Acc support work I think), without backporting the full accelerate support stuff of course.

* Changes needed to get MacOS ARM wheels building again, based on debug work on my fork. [wheel build]

tylerjereddy · 2024-05-21T21:27:06Z

I pushed in a patch that helped deal with the MacOS ARM wheel build issue on my fork, but now of course there's another problem with the new pythran released yesterday. Hopefully I can patch that with less friction.

* Need to pin `pythran` version in a CI job because of a recent new release. [wheel build]

* Increase the Cirrus CI Linux aarch64 job timeout to 72 minutes; it timed out twice in a row via manual restarts.

…o' (scipy#20592)

…) [wheel build] * BUG: fix zipf.pmf for integer k * ENH: increase range for zipfian.pmf for integer k --------- Co-authored-by: Tyler Reddy <tyler.je.reddy@gmail.com> [wheel build]

tylerjereddy · 2024-05-22T04:30:03Z

-            CIBW="DYLD_LIBRARY_PATH=$GFORTRAN_LIB:/opt/arm64-builds/lib delocate-listdeps {wheel} &&\
-              DYLD_LIBRARY_PATH=$GFORTRAN_LIB:/opt/arm64-builds/lib delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}"
+            CIBW="DYLD_LIBRARY_PATH=$GFORTRAN_LIB:$LIB_PATH delocate-listdeps {wheel} &&\
+              DYLD_LIBRARY_PATH=$GFORTRAN_LIB:$LIB_PATH delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}"


@rgommers @h-vetinari This PR is taking shape I think--the full CI matrix (including wheels) is finally passing. The changes inline above are probably the most annoying/controversial ones I had to make manually (they are probably a superset of what is really needed to get Mac ARM wheels passing).

Any remaining objections/concerns for 1.13.1 release? We're at parity with the current list of merged backport-labeled PRs (30 in total now). I have two checklist items left above, but they are basically just updating the relnotes and reading the diff carefully before proceeding.

@mdhaber @dschult @perimosocordiae any more backports that are important for stats or sparse for 1.13.1?

I think if folks are happy I'd try to get 1.13.1 out before branching 1.14.0, with a modest delay likely for the latter, but hopefully not too substantial (I'm blocking off time for release notes work tomorrow).

Any remaining objections/concerns for 1.13.1 release? We're at parity with the current list of merged backport-labeled PRs (30 in total now).

None that I can think of. This is already quite large, so +1 for shipping it soon.

The changes for macOS gfortran should be okay - whatever works there I think is fine in a maintenance branch.

I didn't get to see this earlier, but I think this has all the backports I had in mind.

tylerjereddy · 2024-05-22T04:30:45Z

    platform: linux
    cpu: 2
    memory: 4G
+    timeout_in: 72m


I tried restarting twice; this is probably cheaper than needing to hit restart several times during the rel process.

+1 that makes sense to me, it was always close to the limit.

rgommers · 2024-05-22T07:46:37Z

I'll have a look at the Pythran issue. In the 1.13.x branch the constraint is already "pythran>=0.14.0,<0.16.0",, so the change in this PR to CI config is perfectly fine. On main it sounds like there is something to fix though.

EDIT: okay, no issue after all. It was this:

 ERROR Missing dependencies:
	pythran<0.16.0,>=0.14.0

which is expected is we do a non-isolated build and only pythran 0.16.0 is installed. So everything is fine in main too now.

* Update the SciPy `1.13.1` release notes following substantial backport activity. [skip cirrus]

tylerjereddy · 2024-05-22T16:39:22Z

A new test failure showed up in a Windows CI job (https://github.com/scipy/scipy/actions/runs/9194423643/job/25287826002?pr=20632).

Details

_________________________ TestPoisson.test_fill_space _________________________

self = <scipy.stats.tests.test_qmc.TestPoisson object at 0x000002389148B210>

    def test_fill_space(self):
        radius = 0.2
        engine = self.qmce(d=2, radius=radius)
    
        sample = engine.fill_space()
        # circle packing problem is np complex
>       assert l2_norm(sample) >= radius

C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\scipy\stats\tests\test_qmc.py:938: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\scipy\stats\tests\test_qmc.py:1410: in l2_norm
    return distance.pdist(sample).min()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = array([], dtype=float64), axis = None, out = None, keepdims = False
initial = <no value>, where = True

    def _amin(a, axis=None, out=None, keepdims=False,
              initial=_NoValue, where=True):
>       return umr_minimum(a, axis, None, out, keepdims, initial, where)
E       ValueError: zero-size array to reduction operation minimum which has no identity

C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\numpy\_core\_methods.py:48: ValueError
============================== warnings summary ===============================
stats/tests/test_qmc.py::TestUtils::test_geometric_discrepancy_mst_with_zero_distances
  C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\scipy\stats\tests\test_qmc.py:232: UserWarning: Sample contains duplicate points.
    assert_allclose(qmc.geometric_discrepancy(sample, method='mst'), 0.5)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ===========================
FAILED stats/tests/test_qmc.py::TestPoisson::test_fill_space - ValueError: zero-size array to reduction operation minimum which has no identity

I'm going to restart that now and hope it is a fluke (cc @tupui since it is qmc, but I don't see any reports of it flaking in our issue tracker).

Beyond that, I read through the full diff again and didn't see anything I didn't expect. The rendered versions of the 1.13.1 and 1.13.0 (also modified here) release notes don't seem to have any major issues.

tylerjereddy · 2024-05-22T17:32:04Z

Regular CI is green again, proceeding with 1.13.1 release process.

tylerjereddy · 2024-05-22T17:32:48Z

We have a 1.5:1 commit:backport PR ratio, which is a bit high, but I'd rather not play with commit history/fire.

tylerjereddy · 2024-05-22T20:09:12Z

somehow a Cirrus job still timed out at an hour or so during the rel process.. restarting it is I guess.

dschult and others added 20 commits May 1, 2024 20:21

align dok.pop() to dict.pop() case with no default (scipy#20322)

42de094

DOC: optimize: fix wrong optional argument name in `root(method="lm")…

e5666b6

…`. (scipy#20401)

DOC: add missing deprecations from 1.13.0 release notes

eb8e3a9

[doc only]

MAINT/DOC: fix syntax in 1.13.0 release notes (scipy#20437)

1541f68

DOC: remove spurious backtic from release notes

85c9e29

[doc only]

BUG: linalg: fix ordering of complex conj gen eigenvalues

3687a97

This was reported to cause test failures under windows + MKL in conda-forge cf scipy#20471

TST: interpolate: bump test tolerance in TestRGI::test_derivatives

3c3ac73

A slight tolerance violation was reported on conda-forge in scipy#20472

TST: interpolate: bump test tolerance in TestInterpN

5006ef1

A small tolerance violation reported on conda-forge in scipy#20472

TST: sparse/linalg: bump tolerance for an svds test

3ec0606

tol violation observed on conda-forge on win+blis; suggested in scipy#20474 (comment)

TST: compare absolute values of U and VT in pydata-sparse SVD test

768f46b

BUG: fix spherical_in for n=1 and z=0 (scipy#20527)

a2373cb

BLD: Fix error message for f2py generation fail (scipy#20530)

c7b7322

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>

update openblas to 0.3.27

57e3699

BUG: special: handle uint arrays in factorial{,2,k} (scipy#20607)

a9afa6f

BUG: spherical_in old patch

adaa430

* Apply "old version" of the patch provided at scipygh-20527

DOC: update 1.13.1 relnotes

de2a454

* Update the SciPy `1.13.1` release notes following a series of backports. [skip cirrus]

tylerjereddy added the maintenance Items related to regular maintenance tasks label May 2, 2024

tylerjereddy added this to the 1.13.1 milestone May 2, 2024

tylerjereddy requested review from andyfaff, ev-br, larsoner, perimosocordiae, person142, peterbell10, rgommers and steppi as code owners May 2, 2024 22:27

rgommers and others added 6 commits May 15, 2024 13:38

BUG: value_indices unfixed types

5c770e8

* Fixes scipygh-19423 * Add a few more `case` statements to account for the (i.e., Windows) data types that don't have a fixed width, and add a regression test. [skip circle] [skip cirrus]

MAINT: release branch failures

7db39e8

* attempt to deal with scipygh-20576

MAINT: import fixes

cec68d7

* some minor import fixes following a large series of backports

Revert "MAINT: stats: update boost to improve skewnorm.ppf (scipy#2…

f1575a8

…0643)" This reverts commit ee6203d.

MAINT: import fixes

870e5b4

* more import cleanups after backport activity to make the linter happy

thalassemia and others added 3 commits May 16, 2024 11:47

TST: Adapt to __array__(copy=True)

ce91a92

[skip circle]

CI: MR 20632 revisions

023b0fb

* attempt to fix `M1 test - openblas` MacOS ARM CI job based on some `gfortran` shims applied to a similar job on `main`

tylerjereddy mentioned this pull request May 19, 2024

MAINT, TST: two types of failures observed on maintenance/1.13.x branch #20576

Closed

CI: PR 20632 revisions [wheel build]

aa32933

* This is an empty commit to test the wheel building in the above PR. [wheel build]

tylerjereddy mentioned this pull request May 20, 2024

DEBUG: wheel build debugging tylerjereddy/scipy#77

Open

MAINT: PR 20632 revisions [wheel build]

e7abaf1

* Changes needed to get MacOS ARM wheels building again, based on debug work on my fork. [wheel build]

tylerjereddy and others added 4 commits May 21, 2024 15:29

MAINT: PR 20632 revisions [wheel build]

95c50a4

* Need to pin `pythran` version in a CI job because of a recent new release. [wheel build]

CI, MAINT: PR 20632 revisions

c9d8613

* Increase the Cirrus CI Linux aarch64 job timeout to 72 minutes; it timed out twice in a row via manual restarts.

MAINT: stats.wilcoxon: fix bug with Ndim>1, shape[axis]>50, NaN, 'aut…

506cbeb

…o' (scipy#20592)

BUG: stats: Fix zipf.pmf and zipfian.pmf for int32 k (scipy#20702…

11e99ba

…) [wheel build] * BUG: fix zipf.pmf for integer k * ENH: increase range for zipfian.pmf for integer k --------- Co-authored-by: Tyler Reddy <tyler.je.reddy@gmail.com> [wheel build]

tylerjereddy commented May 22, 2024

View reviewed changes

DOC: PR 20632 updates

1a00d48

* Update the SciPy `1.13.1` release notes following substantial backport activity. [skip cirrus]

tylerjereddy merged commit 2eb8e1b into scipy:maintenance/1.13.x May 22, 2024

tylerjereddy deleted the treddy_prep_1_13_1_backports branch May 22, 2024 17:33

Uh oh!

Conversation

tylerjereddy commented May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylerjereddy commented May 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylerjereddy commented May 19, 2024

Uh oh!

tylerjereddy commented May 19, 2024

Uh oh!

tylerjereddy commented May 21, 2024

Uh oh!

tylerjereddy May 22, 2024

Choose a reason for hiding this comment

Uh oh!

rgommers May 22, 2024

Choose a reason for hiding this comment

Uh oh!

mdhaber May 22, 2024

Choose a reason for hiding this comment

Uh oh!

tylerjereddy May 22, 2024

Choose a reason for hiding this comment

Uh oh!

rgommers May 22, 2024

Choose a reason for hiding this comment

Uh oh!

rgommers commented May 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylerjereddy commented May 22, 2024

Uh oh!

tylerjereddy commented May 22, 2024

Uh oh!

tylerjereddy commented May 22, 2024

Uh oh!

tylerjereddy commented May 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

17 participants

tylerjereddy commented May 2, 2024 •

edited

Loading

tylerjereddy commented May 15, 2024 •

edited

Loading

rgommers commented May 22, 2024 •

edited

Loading