For macOS wheels, I'm currently running into a documented issue with delocate that prevents bundling of libyaml into the wheel because PyYAML uses a 'top-level' _yaml extension module rather than a package structure.

Thanks for starting this work! I'd absolutely recommend cibuildwheel, saves you a lot of reinventing the wheel!

@nsoranzo thanks, though I've already reinvented most of the wheel at this point with manylinux and pure-Python wheels succeeding. I'm not sure if cibuildwheel would alleviate the delocate issue unless it patches it in some way.

To clarify on the delocate issue, delocate-wheel doesn't like that the _yaml extension isn't part of a Python package, and so simply ignores it and fails to bundle in libyaml.

It looks like there is an open PR to fix this (matthew-brett/delocate#39).

Looking good so far at a glance, but a couple things:

• I think you're still going against a dynamically-linked libyaml, which is probably bad. For Linux wheels, check with readelf -d (path to _yamlXXX.so)- if you see any (NEEDED) entries to anything but libc, it's still trying to link dynamically. For Mac ... I have no idea.
• If it's easier to move the extension to a yaml subpackage, go for it- nothing but pyyaml itself should be messing with that

If it's easier to move the extension to a yaml subpackage, go for it- nothing but pyyaml itself should be messing with that

Are you sure there's not a concern for messing with any code out there that's doing import _yaml? I know that's absolutely not part of the public API, but since this package is such a widely used dependency, I would guess that re-namespacing to yaml._yaml would break more than a few things. The other devious option would be to make _yaml the package name itself and import C-extension stuff into its __init__.

I don't personally have any issue with re-namespacing it- I've always hated the fact that the extension was in a different namespace instead of contained beneath the main package. It's pretty clear from the package name that it shouldn't be consumed externally, and if, in the worst case, it causes a lot of problems, we can always backpedal and create a stub package for _yaml (which I think is what you're saying as well). But I wouldn't do that until the pitchfork-wielding masses arrive. ;) Depending on how the efforts to package the wheels works out, we might want to do it as a 6.0 anyway, just to give folks an easy upper-bound to pin to if that causes problems.

Done, via

grep --null -lr " _yaml" tests lib lib3 | \
    xargs -0 -L1  gsed -i 's/ _yaml/ yaml._yaml/g'

Looks like there is now an issue Cythonizing (can't find yaml_parser_t from yaml.h, among other things), so I probably neglected to change something. Will check back later.

Did some Googling for import _yaml- there are definitely folks doing it to do things like check for the presence of the C ext, check its version, and play with its parser internals (for reasons I don't understand at a glance). I'm still not sure we shouldn't just break it anyway, but if not, I'd still personally prefer a stub compatibility package to having the extension itself live in the site-packages root.

Not exactly sure what the issue is here because I'm able to cythonize -i ext/_yaml.pyx just fine. Maybe some of the various Extension stuff happening in setup.py.

Still need to re-work --with-libyaml back into setup.py, but slimmed that file down for now to make things simpler for the moment.

A few considerations for compatibility (they're generally positive things):

The front page of the PyYAML docs give the example python setup.py --with-libyaml install. That's a strong enough reason to retain --with-libyaml and --without-libyaml when installing this way.

pip also advertises its --install-option to pass arguments to setup.py. For example, you might expect this to work:

$ pip install --install-option='--with-libyaml' PyYAML
$ pip install --install-option='--without-libyaml' PyYAML

The blessing in disguise is that the above two commands don't work using the PyPI 5.3.1 PyYAML. I'm not exactly sure why, but probably has to do with the various hacks currently used in setup.py and distutils.

Instead, the straightforward behavior when installing from pip is to use the libyaml bindings if libyaml is found, or otherwise skip them. (setup.py goes out of its way to check for this.) You can see this easily this by running:

pip install --quiet --disable-pip-version-check PyYAML
python -c 'import pyyaml; print(pyyaml.__with_libyaml__)

...first in a python:3-slim Docker container and then in a python:3-stretch, where the results will be False and True, respectively.

So, besides the fact that this PR puts _yaml into yaml._yaml (and that can possibly be monkey-patched still), the only "incompatibility" I see (and it's a huge stretch to call it that), is:

• Previously: user on Ubuntu box calls pip install PyYAML without libyaml present. This will install pyyaml without libyaml bindings
• Now: if manylinux wheels are distributed and the same user goes to install, they will implicitly use the libyaml bindings because libyaml is shipped with the wheel. If they really want to disable this, they should use pip's --no-binary flag.