Skip to content

Workflow : Add nightly release of NumPy in linux workflows #13992

Merged
rgommers merged 15 commits intoscipy:masterfrom
V0lantis:workflow/numpy_nightly_release
May 8, 2021
Merged

Workflow : Add nightly release of NumPy in linux workflows #13992
rgommers merged 15 commits intoscipy:masterfrom
V0lantis:workflow/numpy_nightly_release

Conversation

@V0lantis
Copy link
Copy Markdown
Contributor

@V0lantis V0lantis commented May 5, 2021

Reference issue

This is a follow up to my PR #13876 which was failing once we merged it into master. I tested it on local and on my own fork and it seems to be working, at least I hope. You can found the workflow here

What does this implement/fix?

This PR takes the nightly release of Numpy to have the latest changes.

EDIT :
@BvB93 can you look to the # type: ignore[attr-defined] statement for TestCase import (for example in scipy/optimize/_trustregion_constr/tests/test_qp_subproblem.py). I had the following errors :

scipy/optimize/tests/test_differentiable_functions.py:3: error: Module 'numpy.testing' has no attribute 'TestCase'  [attr-defined]
scipy/optimize/tests/test_minimize_constrained.py:7: error: Module 'numpy.testing' has no attribute 'TestCase'  [attr-defined]
scipy/optimize/tests/test_hessian_update_strategy.py:4: error: Module 'numpy.testing' has no attribute 'TestCase'  [attr-defined]
scipy/optimize/tests/test_constraints.py:3: error: Module 'numpy.testing' has no attribute 'TestCase'  [attr-defined]

I wonder If it comes from numpy?

@BvB93
Copy link
Copy Markdown
Contributor

BvB93 commented May 5, 2021

Yup, good call. Turns out I missed two objects in np.testing.
numpy/numpy#18919 should fix the issue.

@tupui tupui added the CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure label May 6, 2021
@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 6, 2021

Thanks @V0lantis, and thanks for the quick fix @BvB93.

The numpy nightlies are built once a week, on Friday night typically, so a build which includes the fix will show up at https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/ on Saturday. Easiest way to check is to use the version hash (everything after .g in the wheel name, don't ask me why the g is there ....). @V0lantis can you update this PR on Saturday, removing the # type: ignore's? And then I think this is good to merge.

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 6, 2021

I have made the changes and tested upon the commit from @BvB93. it works. I pushed and next Saturday I will merge master to trigger a new workflows.

@andyfaff
Copy link
Copy Markdown
Contributor

andyfaff commented May 6, 2021

Why is the mypy test being run before the actual test run itself? I'd like to see it the other way around so that we can see any test failures first, any typing failures are less consequential.

In addition, 3.10 has now been released to beta. This CI run was originally written to identify issues with as-yet unreleased Python. Therefore, I think we need to amend the "Nightly" Action item to run off Python 3.10. If this is going to cause issues with the mypy run, then either we need to create a new Action, or move the mypy run to another Action item.

@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 7, 2021

Why is the mypy test being run before the actual test run itself? I'd like to see it the other way around so that we can see any test failures first, any typing failures are less consequential.

Probably because mypy is >10x faster to run. Both need to pass, so it doesn't matter much.

In addition, 3.10 has now been released to beta. This CI run was originally written to identify issues with as-yet unreleased Python. Therefore, I think we need to amend the "Nightly" Action item to run off Python 3.10. If this is going to cause issues with the mypy run, then either we need to create a new Action, or move the mypy run to another Action item.

+1 for upgrading to 3.10. It's orthogonal to this PR, but if GitHub Actions has 3.10 available it's a one-liner change. Can you try to include it here @V0lantis? If it doesn't work easily, let's do it in a separate PR.

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 7, 2021

+1 for upgrading to 3.10. It's orthogonal to this PR, but if GitHub Actions has 3.10 available it's a one-liner change. Can you try to include it here @V0lantis? If it doesn't work easily, let's do it in a separate PR.

Yeap, going to try this

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 7, 2021

Why is the mypy test being run before the actual test run itself? I'd like to see it the other way around so that we can see any test failures first, any typing failures are less consequential.

In addition, 3.10 has now been released to beta. This CI run was originally written to identify issues with as-yet unreleased Python. Therefore, I think we need to amend the "Nightly" Action item to run off Python 3.10. If this is going to cause issues with the mypy run, then either we need to create a new Action, or move the mypy run to another Action item.

Unfortunately, we cannot because numpy still doesn't have a release for Python3.10 :

ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
ERROR: No matching distribution found for numpy
Error: Process completed with exit code 1.

You can check the workflow here

Maybe this should be added in the Nightly release @rgommers ?

@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 7, 2021

Ah yes, let's open a separate issue for upgrading the nightly to py3.10, and then work on first getting NumPy nightly wheels for 3.10 up. Not difficult, but it's an extra CI job there I think.

@andyfaff
Copy link
Copy Markdown
Contributor

andyfaff commented May 7, 2021

The nightly CI used to build numpy from source, I don't think it took too long.

@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 7, 2021

Ah yes, going back to building from source should be okay as well. Indeed much faster than building scipy itself, so not a big issue.

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 7, 2021

Ok so should I do an inplace build of numpy in the Nightly python job ?

@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 7, 2021

No, a regular pip install . is preferred. Or, given you don't have a numpy clone yet:

pip install https://github.com/numpy/numpy/archive/refs/heads/main.zip

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 7, 2021

pip install https://github.com/numpy/numpy/archive/refs/heads/main.zip

This doesn't work, so I made a workaround. It is not pretty nor fast but it works fine. I tried with
wget -r -np -R "index.html*" --cut-dirs=6 https://github.com/numpy/numpy/archive/refs/heads/main.zip but it added a lot of overhead so I went back to this solution which works fine.

The failure with wget laid in the version of numpy with wget was :

_get_build_requires
2046
      self.run_setup()
2047
    File "/tmp/pip-build-env-15otxrx6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 248, in run_setup
2048
      super(_BuildMetaLegacyBackend,
2049
    File "/tmp/pip-build-env-15otxrx6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 142, in run_setup
2050
      exec(compile(code, __file__, 'exec'), locals())
2051
    File "setup.py", line 50, in <module>
2052
      MAJOR, MINOR, MICRO = FULLVERSION.split('.')[:3]
2053
  ValueError: not enough values to unpack (expected 3, got 1)

@rgommers
Copy link
Copy Markdown
Member

rgommers commented May 8, 2021

Okay I was being lazy, somehow the right invocation isn't easy to find. You need to use git+https://github.com/.... says https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch.

sudo apt-get install -y --no-install-recommends python3.9-dev python3.9-distutils python3.9-venv
python3.9 -m pip install --upgrade pip setuptools
sudo apt install -y --no-install-recommends python3.10-dev python3.10-distutils python3.10-venv
curl -O https://bootstrap.pypa.io/get-pip.py && python3.10 get-pip.py && rm get-pip.py
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if this is necessary, it's a (temporary I guess) issue with the Python 3.10 that GitHub currently provides. Maybe add a comment that says that and points to this PR. We should be able to remove this later on.

@V0lantis
Copy link
Copy Markdown
Contributor Author

V0lantis commented May 8, 2021

Ok so for me, it ready to be merged

Copy link
Copy Markdown
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All green, in it goes! Thanks @V0lantis

@rgommers rgommers merged commit 0278534 into scipy:master May 8, 2021
@rgommers rgommers added this to the 1.7.0 milestone May 8, 2021
@V0lantis V0lantis deleted the workflow/numpy_nightly_release branch May 10, 2021 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants