Is python setup.py build supposed to work? I thought we're doing pip install -e . now.

I was under the (possibly mistaken?) impression that either should still work, and that pip install -e . was the replacement for python setup.py develop rather than python setup.py build. Or at least that there's some way to build without being required to install into the current environment?

Either way I think it's correct to specify this as a setup-time dependency, right?

Related question that perhaps should be included in this PR: shouldn't the installation docs page also now list extension-helpers? I admit I'm not clear now that I think about it whether extension-helpers is a setup-and-install dependency or just a build dependency, but either way it should probably be somewhere on that page.

Either way I think it's correct to specify this as a setup-time dependency, right?

yes, I think so.

Oh and if it's not clear from my statement above: python setup.py build does seem to work as long as the extension-helpers have already been installed. I guess the question above is whether it should be expected to in the future...

The reason pyproject.toml was introduced is precisely because setup_requires is not able to install dependencies before setup.py is run. This is why for astropy-helpers we had to do all the messing around with the bootstrap file and the submodule. So adding extension-helpers to setup_requires does not achieve anything as you found (so 👎 on the current PR). To run any setup.py command directly, you now need to make sure you have build dependencies installed - not just extension-helpers but also Numpy, Cython, and jinja2. This is expected behavior and consistent with the fact that using tools like pip is now the recommended way as opposed to using setup.py commands.

I'm not sure what your use case is for running build on its own here, but I think the pip equivalent which builds the package without installing it is pip wheel. If you think running build directly is a common use case (I don't remember the last time I ran it) we could always add a helpful message like for test and build_docs? (either redirecting to pip wheel or letting the user which dependencies are needed first)

+1 to what @astrofrog said.

I still contest that @eteq's desire to not install things is not a usecase that's common or one that needs first class support (or one that we should be putting effort into supporting).

But it seems like a pretty serious impediment that the user has no way to know where they're supposed to be getting this from

Just about this point, it's completely ok to tell users they can pip/conda install extension-helpers - this is different to astropy-helpers which was never expected to be installed by users directly.

First, re:

So adding extension-helpers to setup_requires does not achieve anything as you found

and

Just about this point, it's completely ok to tell users they can pip/conda install extension-helpers - this is different to astropy-helpers which was never expected to be installed by users directly.

I see your points here. I don't particularly mind that the users have to install this, I'm concerned that it's not obvious how they would know that. I think in my case the real problem here is that the setup_requires being in setup.cfg means I looked there and then stopped because I thought I'd found the list, and forgot that now I should be looking in pyproject.toml.

But that implies an alternate solution, which is to remove setup_requires entirely, since the "canonical" requirement for building is now in the pyproject (which I have no problem with, I just didn't think to look there first). Or even replacing it with a comment that says "look in pyproject.toml, not here. Is there any downside to that which I'm missing? If not, I'm happy to update this PR to do that instead.

Now separately, @astrofrog's

I'm not sure what your use case is for running build on its own here

and @Cadair's

I still contest that @eteq's desire to not install things is not a usecase that's common or one that needs first class support (or one that we should be putting effort into supporting).

There are a several different things here:

1. Many people who want to build a package that aren't doing it for the first time are used to doing python setup.py build if there's a setup.py. That won't change for some time (regardless whether that's desirable or not, it's true)

Fortunately that one is well-addressed by @astrofrog's proposal to provide a clear re-direct like what we are currently doing with test. There is the question of exactly what that should re-direct to, though (see 3 below for a bit more).

2. When doing anything that tests or alters something that's not just regular python code, I usually want as few things between me and the result of the build process as possible. With pip it's hard to tell for various reasons (i.e. build logs from other packages are mixed in, you have to do -v to get build logs at all, etc). That used to be simple and straightforward via build, but now there's several layers in between.

I'm fine with the answer to this one being "if you're that deep in you need to install the pyproject.toml requirements", which is perfectly fine as long as we mention that in the install docs (currently pyproject.toml is basically listed as an "implementation detail for pip" rather than the canonical list of requirements for building, but that's an easy wording change).

Unfortunately this solution conflicts with the solution to 1... I honestly don't care at all if it's not python setup.py build but I want some way to just get the build without having to do an install or any other pip business.

3. When I want to update something in an environment I frequently want to "try before I buy". That means having a place I can build into without installing to check that it hasn't broken something. I'm certain I'm not alone in that.

The pip wheel . solution doesn't provide an obvious fix for that. It yields a wheel, which is not really something equivalent to what build gives - something I can easily cd into and treat as though it's the thing I would install if I were to run install. Again I don't really care if it's python setup.py build or a tox invocation or something else, but I think it's a very real use case that people expect to be able to build some source code they have without being required to install it at the same time...

Looking at the above, I suspect this discussion will go on for some time and is a bit peripheral to the immediate point of this PR, @astrofrog and @Cadair.

Perhaps we could shift the second part of that discussion to a separate issue and just do the setup_requires change in this PR? (Or close this if you both think neither of the changes I've suggested for this PR are warranted - i.e. the part of my above comment before "Now separately ... ")

At the very least, there is no harm in putting an extra comment into setup.cfg for the benefit of contributors ("normal" users are unlikely to look there) who are not following the APE17 business closely.

I'm confused:

Perhaps we could shift the second part of that discussion to a separate issue and just do the setup_requires change in this PR? (Or close this if you both think neither of the changes I've suggested for this PR are warranted - i.e. the part of my above comment before "Now separately ... ")

I see only one change in the PR? What other change were you thinking of?

As I mentioned before, the change to setup_requires really doesn't solve anything as you've seen, so I'm very much against doing it. There is no way to auto-install build dependencies for the user if you aren't using pip/pyproject. However, as I mentioned before, we could easily just check at the top of the setup.py file if the build dependencies are available and give a nice error for the user if not, telling them what packages to install.

And to be clear, I'd be ok with just removing setup_requires altogether from setup.cfg

Cleared milestone as this is not critical for 4.1 - though the larger issue definitely is something we need to consider: I know @taldcroft and I also have been struggling with regaining the ability to quickly test things. It would be very good to try to ensure that is possible in the coming months, but I think we need a separate issue that can collect what breaks, what is best practice, etc. (the suggestion of working within a tox environment that was made recently was one that seemed promising). Just to be sure: I love having gotten rid of astropy_helpers, if only for not having it pop up unexpectedly in my PRs!

Hi humans 👋 - this pull request hasn't had any new commits for approximately 5 months. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

Merging this will have zero benefit because setup_requires will only be honored after the setup.py file has run, and that requires extension helpers to already be installed. So I still strongly favour closing this.

In fact I will just close it 😬

(I will open a PR with an alternative approach)