Cool. I think pip supports installing from a directory, so we could build a Cython wheel and let pip install that for the test environment.

@scoder did you mean in CI? For CI, I tend to build dists first and then, use them in tests so that the following job could publish what's actually been tested. Let me know if you'd like a PR or pointers.

allowing pip to use isolated build environments, and at that stage it will want to download Cython via pypi rather than use the local version that we want to test.

You can add --no-index --find-links to pip wheel (or pip install for that matter) and that should avoid hitting PyPI. Additionally, python -Im build has a --no-isolation flag which can let you invoke hooks if you pre-provision the deps. There's also pyproject_hooks that may be useful, perhaps.

Oh, and the downstreams have gpep517, pyproject-rpm-macros and dh-python/pybuild-plugin-pyproject that help them deal with the PEP 517 / 660 interface.

Let's also invite @abravalheri to get an opinion from the setuptools upstream.

allowing pip to use isolated build environments, and at that stage it will want to download Cython via pypi rather than use the local version that we want to test.

You can add --no-index --find-links to pip wheel (or pip install for that matter) and that should avoid hitting PyPI. Additionally, python -Im build has a --no-isolation flag which can let you invoke hooks if you pre-provision the deps. There's also pyproject_hooks that may be useful, perhaps.

I forgot one more trick — you can have a constraints file and set the PIP_CONSTRAINT variable to that — it leaks into the underlying isolated call of pip install. The same goes for PIP_NO_INDEX and PIP_FIND_LINKS.

Let's also invite @abravalheri to get an opinion from the setuptools upstream.

Last year we did something in setuptools-rust to add support for pyproject.toml but we followed a different approach and avoided patching setuptools altogether. I recon something very similar can be done for Cython.

For example, considering the following already exists in setuptools:

1. A hook via the setuptools.finalize_distribution_options entrypoint that allows setting the attributes corresponding to the setup() keywords.
2. Support for .pyx sources in Extension objects when Cython is installed (ref1, ref2).

Something like the following could be implemented:

# Draft based on setuptools-rust

if sys.version_info[:2] >= (3, 11):
    import tomllib
else:
    import tomli as tomlib


def pyprojecttoml_config(dist: Distribution) -> None:
    try:
        with open("pyproject.toml", "rb") as f:
            cfg = tomllib.read(f).get("tool", {}).get("cython")
    except FileNotFoundError:
        return None

    if cfg:
        existing = dist.ext_modules or []
        new = map(_create, cfg.get("ext-modules", []))
        dist.ext_modules = [*existing, *new]


def _create(config: dict) -> Extension:
    from setuptools.extension import Extension

    # PEP 517/621 convention: pyproject.toml uses dashes
    kwargs = {k.replace("-", "_"): v for k, v in config.items()}
    return Extension(**kwargs)

# Then, an entrypoint needs to be defined in the Cython distribution:
#
# [project.entry-points."setuptools.finalize_distribution_options"]
# cython = "Cython.whichever_module:pyprojecttoml_config"

That said, no specific code seems to be necessary to handle something like the following definition:

[[tool.cython.module_list]]
name = "a"
sources = ["a.pyx"]

... and from the setuptools point of view, this is probably likely to be implemented some day in the form of:

[[tool.setuptools.ext-modules]]
name = "a"
sources = ["a.pyx"]

(in fact we reached a point in the development of setuptools that we can start working on a PR for that¹).

What is unlikely to be supported directly in setuptools is something like the following:

[tool.cython]
cythonize = ["a.pyx", "b.pyx", "c.py"]

But this could be implemented following the methodology discussed above (setuptools.finalize_distribution_options). The difference is that for each .pyx an extension object with a corresponding name would be created.

/crossref pypa/setuptools#4568

Thanks @abravalheri - one of the things I was initially worried about was that it seemed to be (mostly) implementing a feature that would be better off in setuptools, and that the "cython" part of this was comparatively small.

It sounds like I should wait for the setuptools PR, and then investigate what's missing. And try to implement it with the less intrusive finalize_distribution_options if possible.

@joshua-auchincloss this ain't for hatchling but perhaps you'd be interested in checking out the interface for building Cython extensions. Any feedback?

I'm not sure this is worth anyone reviewing at this point - most of it will hopefully be removed fairly soon.

Simple use-cases can now be tried out with setuptools>74.1.
For example the following should be possible to write:

[build-system]
requires = ["setuptools>74.1", "cython"]
build-backend = "setuptools.build_meta"

[project]
name = "something"
version = "0.1"
description = "description"

[tool.setuptools]
ext-modules = [
   {name = "a", sources = ["a.pyx"]},
   {name = "b", sources = ["b.pyx"]},
]

Feedback targeting setuptools functionality should be directed at the pypa/setuptools repository.

More complex use cases (e.g. passing flags and configuration parameters to cythonize) are not covered in setuptools, and may be implemented by Cython (if the maintainers are interested¹) using the setuptools.finalize_distribution_options hook as summarised in #6338 (comment).

Thanks for merging that @abravalheri

I think what I'll do in the near future is add some documentation for it in Cython so that people know start using the declarative version.

I'll have to think about where because our "source files and compilation" docs page already seems to big and unwieldy to me so I don't really want to just dump more information in it.

I'm definitely willing to expand it a bit with Cython-specific additions as you suggest. Possibly worth seeing what gets requested first though.

I'll close this PR because I think it's completely superseded.