Improve docs on running packages#2494
Conversation
Thanks. will do. |
|
So now the recommended commands build all dependencies too, and I run into some error in kiwisolver (when specifying all dependencies - besides requests - for obspy in the recipe), see log below. Note that at the end there are message "successful" and "error", so the whole thing seems confused a bit. If I comment out enough dependencies so that kiwisolver doesnt get built, I run into errors building numpy (pyodide_build doesnt find Python.h for some reason). So on this branch I can only get obspy to build if I remove all dependencies from the yaml, which is kinda defeating the purpose of building with all dependencies so that the created From a packaging point of view it certainly is ideal to also build dependencies, but I think the old way to really only build that one package and try it mixed into the stable release has it's charm of simplicity too.. might be worth considering having that option too? Maybe offer a build command that..
In any case, I ended up again only building obspy and tossing it in the unpacked stable release and modifying the package.json. Status seems the same as on main, that simple C extension I tested before works and the complex one fails with the same error. |
| If the build succeeds you can try to load the package by | ||
|
|
||
| 1. If you haven't built Pyodide from source, download the | ||
| [most recent Pyodide release](https://github.com/pyodide/pyodide/releases) and extract it into the `dist` directory |
There was a problem hiding this comment.
I find that step 1) confusing. I don't think it belongs here under 3. Building the package and investigating issues. Everything in this part is about building the package so that it gets dropped into pyodide/dist with everything in there to run it. If I where to unpack the stable release in that directory, I would overwrite a) all the dependencies that were just compiled (if it works) with the versions from the tarball and b) also the package.json that would then again lack the new package.
If going through with forcing the new package build to also rebuild all dependencies, I think this should only list steps 2.-4.
But I still think a workflow were really only the new package is built and just using everything else from the stable release would be the easiest solution for people to try and build their package, if all dependencies are already in the stable release.
There was a problem hiding this comment.
a workflow were really only the new package is built and just using everything else from the stable release would be the easiest solution
We are working toward this. Hopefully the next major release will include this feature.
|
Thanks so much for your help with this @megies!
The issue is that we have more work to do before we support out of tree builds. Currently this only works if you get lucky. Let me explain. The issue is that we have to ship a cross-build environment. Cross-build environments
These problems are because you don't have a cross build environment. Python does not have proper support for cross compilation. When you build the Python interpreter, it works out a bunch of constants of the underlying architecture and OS -- pointer size, float sizes, does the OS have When building a package, Python has static assertions that double check that your We don't currently ship our There is yet again a problem with numpy,
The resulting numpy is a "cross-build" numpy: it runs on the build machine but constructs packages for the target machine. It cannot be used to build downstream packages for the build machine. We do something similar with scipy and we should do something similar with the In total, we need crossbuild versions of: Python itself, numpy, scipy, cffi, pycparser, pythran, and cython. Without this crossbuild environment, you can't build C extensions. But we don't currently ship such an environment so you have no choice but to build the Python interpreter and then at least those numpy, scipy, cffi, and pycparser (pythran and cython are build-time only they have no runtime behavior). |
You're losing me whenever wasm comes into play, don't know anything about it really.. So, what you're saying is, you can't really easily build packages that have (anything but simplemost) C extensions without compiling Python from source? I mean compiling C from source isn't a big deal but, if it's more complicated than that, it's probably not worth meddling with local builds and just rely on test builds in PR relying on your CI setup, I guess? If that's the case, I think docs just point out that there are problems and that it's best to just test build in CI. I couldn't find the artifacts and where to download them from the CI run, maybe there could be info on this in the docs too? I am assuming the whole build gets tar'ed up and uploaded as an artifact? |
It is possible if we make improvements to our build system but we need to do more work to enable this. I am looking into it. We really want to enable out of tree builds because eventually we don't want to be responsible for maintaining all the recipes.
They are here: |
|
Oops. I meant to merge this. |
|
Except your review still needs to be addressed... Thanks for flagging this again @megies. |
|
I think if we fixed |
|
Also compare https://twitter.com/ocefpaf/status/1526257610650951680 by @ocefpaf which also ends right at that point where.. when you went to build locally and not in CI, I'd figure 99% of people also would like to try the package out locally (e.g. in a pyscript REPL), instead of just staring at a wheel, pondering if what's inside will actually work =) |
ryanking13
left a comment
There was a problem hiding this comment.
Thanks, @hoodmane. It looks like I forgot to review this :/
Apparently, the current situation of building a single package and testing it is not ideal. But I hope we can make it better gradually soon.
| `test_<package-name>.py`. The tests should look like: | ||
|
|
||
| ```py | ||
| from pyodide_build import run_in_pyodide |
There was a problem hiding this comment.
| from pyodide_build import run_in_pyodide | |
| from pyodide_test_runner import run_in_pyodide |
| vendors tests you can do it like: | ||
|
|
||
| ```py | ||
| from pyodide_build import run_in_pyodide |
There was a problem hiding this comment.
| from pyodide_build import run_in_pyodide | |
| from pyodide_test_runner import run_in_pyodide |
|
Thanks @megies! We should have significant improvements to this workflow coming soon. |
|
Oops, I thought I'd merged @ryanking13's comments... |
|
Thanks @hoodmane and the rest of the crew, I can only imagine how you folks get bombarded with work these days 👍 |
Thanks to @megies for hassling us about this.
@megies can you review this? Note that we adjusted
pyodide_build buildallto fix some bugs. Could you try building your package against this branch with the described workflow and let us know if you run into trouble?