Skip to content

Use nox to manage development#59

Merged
leouieda merged 19 commits intomasterfrom
nox
Oct 27, 2020
Merged

Use nox to manage development#59
leouieda merged 19 commits intomasterfrom
nox

Conversation

@leouieda
Copy link
Member

Nox is a tool that automates setting up virtual environments and running tasks inside them. It's a combination of make and virtual environments. Replace the machinery in our Makefile (which wouldn't work on Windows) with a cross-platform noxfile.py. Since nox handles the virtual environments, the environment.yml only needs to have python and nox. Plus, the environment creation code in the CI is no necessary anymore and we can run tests on multiple python versions locally as well.

Reminders:

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst and the base __init__.py file for the package.
  • Write detailed docstrings for all functions/classes/methods. It often helps to design better code if you write the docstrings first.
  • If adding new functionality, add an example to the docstring, gallery, and/or tutorials.
  • Add your full name, affiliation, and ORCID (optional) to the AUTHORS.md file (if you haven't already) in case you'd like to be listed as an author on the Zenodo archive of the next release.

@leouieda leouieda changed the title Use nox to manage development WIP Use nox to manage development Oct 14, 2020
@leouieda leouieda changed the title WIP Use nox to manage development Use nox to manage development Oct 15, 2020
@leouieda
Copy link
Member Author

I'm keeping the Makefile so that most instructions on the CONTRIBUTING.md still work but it's just a wrapper of the nox commands. We could remove it entirely and run nox instead.

@leouieda
Copy link
Member Author

Now that I think about it, I would probably organize the CI workflows differently:

  • style.yml: stay the same
  • test.yml: just have jobs for running the tests and submitting coverage. No docs builds here.
  • docs.yml: build the docs and if on master or a release, deploy to gh-pages.
  • pypi.yml: deploy to PyPI and TestPyPI.

This way we avoid having multiple duplicate builds of the docs. We could still build on all systems but I think it would be fine to only run it on ubuntu.

@leouieda leouieda requested a review from santisoler October 15, 2020 09:41
@santisoler
Copy link
Member

This looks great @leouieda! It's awesome to be capable of running tests on multiple Python versions locally without too much hassle.

For some reason my computer is skipping the py3.6 test:

nox > Session test-3.6 skipped: Python interpreter 3.6 not found.

Do you know why?

I did notice that even if you set up your conda environment, if you try to run nox without internet connection it fails because cannot install the dependencies. Is there a way to preinstall everything that nox needs beforehand so we don't depend on an internet connection for running the tests?

@santisoler
Copy link
Member

santisoler commented Oct 15, 2020

I did notice that even if you set up your conda environment, if you try to run nox without internet connection it fails because cannot install the dependencies. Is there a way to preinstall everything that nox needs beforehand so we don't depend on an internet connection for running the tests?

Actually there is:

nox --install-only

https://nox.thea.codes/en/stable/usage.html?highlight=offline#skipping-everything-but-install-commands

Buuuut be very careful. If you run nox without the -r option you will undone the previous step and end up with no preinstalled environments. If we remove the Makefile in the future I can certainly see myself breaking my preinstalled environments very often, which will be a pain if I'm a on a place with bad internet connection, or no connection at all.

Maybe we can open an Issue on their repository for this.

Copy link
Member

@santisoler santisoler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some suggestions down here. I'm not entirely convinced about nox, but I like the idea! I think I just need to spend some time with it to get used to it.

@leouieda
Copy link
Member Author

About running offline:

  • The --install-only lets you build the environments only without running tests.
  • I'm hacking on this in Erizo and will copy over some improvements to allow skipping install steps so that you can work offline

The default is making fresh environments but we can reverse that by setting nox.options.reuse_existing_virtualenvs=True. We can always disable on the command line with --no-reuse-existing-virtualenvs. I'll set this on and then we can skip the -r without much fear of nuking the environments.

@leouieda
Copy link
Member Author

Alright, with the new file after 08e23c4 you can now:

  • Reuse environments by default
  • Pass in the skip-install option to any session to skip the install step
  • Run all checks or selected checks by passing arguments to the check session.

Need to add these instructions to the docstring still.

@leouieda leouieda requested a review from santisoler October 19, 2020 10:45
@leouieda
Copy link
Member Author

@santisoler if you could give this a try locally that would be very helpful. See if you like it and if there are any other pain points we can try to resolve

Copy link
Member

@santisoler santisoler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great @leouieda!

I'm linking the simplicity of nox (and writing workflows in Python is much easier for me than writing Makefiles). I added some suggestions, please see if you like them.

I would like to add a few make targets:

  • make nox-install: for creating the virtual environments and installing dependencies (leaving everything ready for offline runs).
  • make nox-clean: for cleaning the .nox directory.
  • make nox-update: for cleaning and then installing the environments. I think this might be useful not only for offline runs but also for updating any old-dated nox environments.

Also, I would like to add a serve session for serving the built docs locally.

Let me know what do you think.

Now conda-forge has nox for 3.9 already
Also add instructions for using the noxfile in the docstring.
@leouieda
Copy link
Member Author

@santisoler I made a few changes:

  • Added instructions to the noxfile.py docstring
  • Added a show option to the docs build that opens the HTML in a web browser. I opted for this instead of serving since it would block the process and the noxfile would just hang. This works just as well and is easily cross-platform using import webbrowser.
  • Added rules to the Makefile for nox-clean and nox-install. The update command is not needed since it would be a clean and install anyway.

What do you think?

@santisoler
Copy link
Member

This is looking great!

* Added instructions to the noxfile.py docstring

I like that we can list all the available targets with nox -l. We definitely should add this to the Contributing instructions once we ditch the make instructions.

* Added a `show` option to the docs build that opens the HTML in a web browser. I opted for this instead of serving since it would block the process and the noxfile would just hang. This works just as well and is easily cross-platform using `import webbrowser`.

Much better than before!

* Added rules to the Makefile for `nox-clean` and `nox-install`. The update command is not needed since it would be a clean and install anyway.

Sure, I was thinking to add the nox-update target calling the nox-clean and then nox-install, but we don't actually need another target to do so, we can use make nox-clean nox-update on a single line.

I think this is ready to go! Nice work @leouieda !

@leouieda
Copy link
Member Author

Thanks @santisoler! Merging this in. I'll work on a follow up to organize the CIs a bit. Only thing I changed is that setuptools_scm doesn't need to be a run time dependency.

@leouieda leouieda merged commit bf24f41 into master Oct 27, 2020
@leouieda leouieda deleted the nox branch October 27, 2020 11:47
leouieda added a commit that referenced this pull request Oct 27, 2020
In #59, we inadvertently reverted the `sed` command in the `deploy`
workflow to the versioneer version. This reintroduces the correct
command from #61.
leouieda added a commit that referenced this pull request Oct 27, 2020
In #59, we inadvertently reverted the `sed` command in the `deploy`
workflow to the versioneer version. This reintroduces the correct
command from #61.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants