Out-of-tree recipe build support #3544

ryanking13 · 2023-02-06T01:10:43Z

Description

Add out-of-tree support of pyodide build-recipes. Thanks to the awesome xbuildenv work by Hood, the required code change was quite small.

I am testing this branch here: https://github.com/ryanking13/pyodide-recipes-mirror

Checklists

Add a CHANGELOG entry

hoodmane

Thanks for working on this @ryanking13!

it builds:

numpy

This was one of my main objectives when working on the out of tree builds project. =)

it fails to build:

scipy

Yeah, that's hopeless, at least until we get a real fortran compiler. Even then it may take signficant work

pyodide-build/pyodide_build/cli/build_recipes.py

pyodide-build/pyodide_build/cli/skeleton.py

pyodide-build/pyodide_build/pypabuild.py

ryanking13 · 2023-02-09T00:57:49Z

Scipy build error seems related to pythran... but I'm not sure.

scipy/interpolate/_rbfinterp_pythran.cpp:82:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type2>())) __type3;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:85:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type6>())) __type7;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:86:29: error: implicit instantiation of undefined template 'std::tuple_element<0, int>'
      typedef typename std::tuple_element<0,typename std::remove_reference<__type7>::type>::type __type8;
                            ^
/workspaces/pyodide-recipes/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__tuple:56:62: note: template is declared here
template <size_t _Ip, class _Tp> struct _LIBCPP_TEMPLATE_VIS tuple_element;
                                                             ^
scipy/interpolate/_rbfinterp_pythran.cpp:90:47: error: subscripted value is not an array, pointer, or vector
      typedef decltype(std::declval<__type3>()[std::declval<__type11>()]) __type12;
                       ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
scipy/interpolate/_rbfinterp_pythran.cpp:93:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type14>())) __type15;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:95:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type17>())) __type18;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:96:29: error: implicit instantiation of undefined template 'std::tuple_element<0, int>'
      typedef typename std::tuple_element<0,typename std::remove_reference<__type18>::type>::type __type19;
                            ^
/workspaces/pyodide-recipes/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__tuple:56:62: note: template is declared here
template <size_t _Ip, class _Tp> struct _LIBCPP_TEMPLATE_VIS tuple_element;
                                                             ^
scipy/interpolate/_rbfinterp_pythran.cpp:100:48: error: subscripted value is not an array, pointer, or vector
      typedef decltype(std::declval<__type15>()[std::declval<__type22>()]) __type23;
                       ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
scipy/interpolate/_rbfinterp_pythran.cpp:130:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type4>())) __type5;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:133:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type8>())) __type9;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:134:29: error: implicit instantiation of undefined template 'std::tuple_element<0, int>'
      typedef typename std::tuple_element<0,typename std::remove_reference<__type9>::type>::type __type10;
                            ^
/workspaces/pyodide-recipes/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__tuple:56:62: note: template is declared here
template <size_t _Ip, class _Tp> struct _LIBCPP_TEMPLATE_VIS tuple_element;
                                                             ^
scipy/interpolate/_rbfinterp_pythran.cpp:138:47: error: subscripted value is not an array, pointer, or vector
      typedef decltype(std::declval<__type5>()[std::declval<__type13>()]) __type14;
                       ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
scipy/interpolate/_rbfinterp_pythran.cpp:144:47: error: subscripted value is not an array, pointer, or vector
      typedef decltype(std::declval<__type5>()[std::declval<__type22>()]) __type23;
                       ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
scipy/interpolate/_rbfinterp_pythran.cpp:175:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type4>())) __type5;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:178:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type8>())) __type9;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:179:29: error: implicit instantiation of undefined template 'std::tuple_element<0, int>'
      typedef typename std::tuple_element<0,typename std::remove_reference<__type9>::type>::type __type10;
                            ^
/workspaces/pyodide-recipes/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__tuple:56:62: note: template is declared here
template <size_t _Ip, class _Tp> struct _LIBCPP_TEMPLATE_VIS tuple_element;
                                                             ^
scipy/interpolate/_rbfinterp_pythran.cpp:183:47: error: subscripted value is not an array, pointer, or vector
      typedef decltype(std::declval<__type5>()[std::declval<__type13>()]) __type14;
                       ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
scipy/interpolate/_rbfinterp_pythran.cpp:213:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type6>())) __type7;
                       ~~~~~~~~~~~~~~~~~^
scipy/interpolate/_rbfinterp_pythran.cpp:216:41: error: no member named 'as_const' in namespace '(anonymous namespace)::pythonic::types'
      typedef decltype(pythonic::types::as_const(std::declval<__type10>())) __type11;

hoodmane · 2023-02-09T01:13:19Z

scipy error

But that's just the first error. There will be many many more if you fix that one.

ryanking13 · 2023-02-09T01:21:03Z

But that's just the first error. There will be many many more if you fix that one.

But we are building scipy in-tree with the same recipe, so if everything works correctly it should be able to build out-of-tree too I think.

ryanking13 · 2023-02-13T00:16:08Z

Okay, I was able to build scipy out-of-tree with the same recipe. The problem was:

pythran 0.12.1 and 0.12.0 are not compatible
pyodide-build 0.22.1 xbuildenv include pythran 0.12.0 in the host site packages.
scipy defines pythran in its pyproject.toml, so it installs latest version (0.12.1) inside the build env.
two differenct versions of pythran conflicts during the build

Opened #3576 for a fix.

ryanking13 · 2023-02-20T06:43:52Z

I think this is ready to be reviewed now. I managed to build core package sets out-of-tree including numpy, scipy, pandas, matplotlib, scikit-learn, scikit-image, Pillow, opencv-python, xgboost.

hoodmane · 2023-02-20T07:38:06Z

Thanks! Will review soon.

ryanking13 · 2023-02-20T07:44:43Z

Thanks! This is on top of #3592.

hoodmane

Thanks for working on this @ryanking13! I'm sorry it took me so long to get around to reviewing it.

pyodide-build/pyodide_build/cli/skeleton.py

hoodmane · 2023-03-21T13:39:45Z

pyodide-build/pyodide_build/out_of_tree/utils.py

        return
    try:
-        os.environ["PYODIDE_ROOT"] = str(search_pyodide_root(__file__))
+        os.environ["PYODIDE_ROOT"] = str(search_pyodide_root(Path.cwd()))


Is this a safe change? It does something pretty different. Could you explain the logic before vs now?

I think there was a bug in the logic before, which caused an error here: #3669.

Previously it was finding the pyodide root from the parents of that file. So if we don't install pyodide-build in a virtual environment inside the project folder, it failed.

…dide into build-recipes-oot

hoodmane

Thanks again @ryanking13 !

ryanking13 added 6 commits January 27, 2023 05:09

Support out-of-tree recipe build

ba9cb35

Handle pyenv shims

bb6b2be

Add pyodide-build module root to PYTHONPATH

94d7d6a

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

8f642e8

Set n_jobs based on the number of host cpu cores

2c97ab4

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

6874715

hoodmane reviewed Feb 6, 2023

View reviewed changes

pyodide-build/pyodide_build/cli/build_recipes.py Outdated Show resolved Hide resolved

pyodide-build/pyodide_build/cli/skeleton.py Outdated Show resolved Hide resolved

pyodide-build/pyodide_build/pypabuild.py Outdated Show resolved Hide resolved

This was referenced Feb 9, 2023

ENH Autodetect the number of cores when invoking pyodide build-recipes #3559

Merged

BLD Remove _pywasmcross entrypoint #3566

Merged

ryanking13 added 17 commits February 14, 2023 02:00

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

2e7e07e

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

f6290cf

Fix issue after merge

2b78598

Revert appending pyodide-build root to PYTHONPATH

598ec0c

Add URL parameter to install_xbuildenv

16b2604

Add test

4628de3

Fix path

7eeb097

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

3212a18

Fix finding recipe directory when in-tree

e367cf6

Merge branch 'xbuildenv-cli' into build-recipes-oot

7c9fa42

Update PYODIDE_ROOT inside install_xbuildenv

346c39c

Merge branch 'xbuildenv-cli' into build-recipes-oot

0bb3a13

Revert changes in initialize_pyodide_root

aea09a8

Merge branch 'xbuildenv-cli' into build-recipes-oot

d563a79

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

d1dbac2

Cleanup

2d455e5

Update changelog

f789ad7

Update opencv-python meta.yaml

6d7db13

ryanking13 changed the title ~~Draft: Out-of-tree recipe build support~~ Out-of-tree recipe build support Feb 20, 2023

ryanking13 marked this pull request as ready for review February 20, 2023 06:41

ryanking13 added 2 commits March 6, 2023 02:46

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

5ad8910

Merge branch 'pyodide:main' into build-recipes-oot

37654ec

rth added this to the 0.23.0 milestone Mar 17, 2023

hoodmane mentioned this pull request Mar 17, 2023

0.23 release #3666

Closed

hoodmane reviewed Mar 21, 2023

View reviewed changes

ryanking13 added 3 commits March 22, 2023 05:30

Merge remote-tracking branch 'upstream/main' into build-recipes-oot

06dc32f

Apply suggestion by Hood

9d78b36

Merge branch 'build-recipes-oot' of https://github.com/ryanking13/pyo…

01a4b3e

…dide into build-recipes-oot

hoodmane approved these changes Mar 22, 2023

View reviewed changes

ryanking13 merged commit b801885 into pyodide:main Mar 23, 2023

ryanking13 deleted the build-recipes-oot branch March 23, 2023 05:01

hoodmane mentioned this pull request Mar 31, 2023

Broken pyodide build /abs/path/to/module in 0.23.0 #3722

Closed

ryanking13 mentioned this pull request Apr 12, 2023

Package OpenBLAS and use OpenBLAS in scipy #3331

Merged

1 task

Uh oh!

Out-of-tree recipe build support #3544

Out-of-tree recipe build support #3544

Uh oh!

Conversation

ryanking13 commented Feb 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

hoodmane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanking13 commented Feb 9, 2023

Uh oh!

hoodmane commented Feb 9, 2023

Uh oh!

ryanking13 commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanking13 commented Feb 13, 2023

Uh oh!

ryanking13 commented Feb 20, 2023

Uh oh!

hoodmane commented Feb 20, 2023

Uh oh!

ryanking13 commented Feb 20, 2023

Uh oh!

hoodmane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hoodmane Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

ryanking13 Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoodmane left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryanking13 commented Feb 6, 2023 •

edited

Loading

ryanking13 commented Feb 9, 2023 •

edited

Loading

ryanking13 Mar 22, 2023 •

edited

Loading