I managed to add some debug to have the explicit command used and it looks like -shared is used but that -shared for emscripten is not doing the thing we need:

[6/168] emcc  -o skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so.p/meson-generated_fast_exp.c.o -sERROR_ON_UNDEFINED_SYMBOLS=0 -shared -fPIC -Wl,--start-group -lm -Wl,--end-group
emcc: warning: linking a library with `-shared` will emit a static object file.  This is a form of emulation to support existing build systems.  If you want to build a runtime shared library use the SIDE_MODULE setting. [-Wemcc]

Adding the lazy_loader package, CI is green (except codecov), I guess this is good enough to mark it as ready for feed-back.

What needs still to be figured out/done:

• what would be a better way to do it without using a meson fork?
• locally (apparently not in the CI for some reason) I do have to remove the space after -L otherwise I get a build error because somehow the -L and its path argument get separated at one point??? Maybe someone can try to see if this is something weird with my setup?
• Cleanup
• other things I may have forgotten

@henryiii if you have any advice about this it would be appreciated.

what would be a better way to do it without using a meson fork?

The one change I see is rth/meson-python@115683f, which disables the native file and hardcodes the cross file location.

The cross file location can be given directly instead, this should be equivalent:

# `pip` instead of `build` works too (for pip<23.1 you need to spell it `--config-settings=` instead of `-C`)
python -m build -Csetup-args=--cross-file=$PYODIDE_ROOT/tools/emscripten.meson.cross

I'm unclear why it would be needed to disable the native file. Its only purpose is to ensure that the Python interpreter used by Meson is the one from the environment in which meson-python is installed. It should not matter, assuming meson and meson-python are indeed installed in the active build env.

Thanks a lot for your feed-back!

This change actually comes from the original attempt at using meson for scipy #3037 from end August 2022. Maybe things have improved since then. I'll try to have a closer look based on your suggestion.

Ah yes, things did improve, with the latest meson-python release I think there should not be a problem anymore.

Looks like that is pretty happy:) It'd be even nicer if there were a build.sh like in conda-forge, or some other way of specifying the build command to be used for any package. Having to use the current meson-debug.patch only to add pip CLI flags is a little unfortunate. Maybe something like that is already in the works?

Config-settings is supported in cibuildwheel, so pyodide-build must be able to handle it too? (Can't check currently, but can soon).

Looks like that is pretty happy:)

Yep 🎉

Having to use the current meson-debug.patch only to add pip CLI flags is a little unfortunate. Maybe something like that is already in the works?

Indeed, this is the first thing I could think of to specify a cross-file without a meson-python fork. I am hoping that someone more knowledgeable than me about the Pyodide tooling will have a better suggestion about how to do this in a nicer way!

By the way, there is a huge benefit if this is possible via meson(-python) environment variable (like CMAKE_TOOLCHAIN for CMake). Then cibuildwheel could set the variable and meson-python would support producing pyodide wheels out of the box, like it does with setuptools and scikit-build, rather than requiring custom config added to every project.

By the way, there is a huge benefit if this is possible via meson(-python) environment variable (like CMAKE_TOOLCHAIN for CMake)

That does not sound too appealing. Every packaging tool has its own environment variables, and silently handling all those inside meson-python makes it harder to figure out what is happening. Meson supports some of the bog-standard ones like CC and CFLAGS, but has a documented policy against using env vars in general: https://mesonbuild.com/Contributing.html#environment-variables.

Example: conda-forge uses a MESON_ARGS env var for all projects using Meson. We already had a discussion there, and it's a one-liner to turn that into pip/build CLI arguments (see, e.g., here for scipy).

It's a little more work, but the linked Meson policy is correct in my experience - once you have it set up, it's much nicer to have CLI args than env vars, because you can more easily see what is going on in a build.

The problem is cibuildwheel --platform pyodide (or emscripten, I forget was was chosen) should work without having to add custom configuration for meson projects. cibuildwheel can set environment variables (MESON_PYTHON_CROSSCOMPILE, for example), but it can't add command line flags that are not supported by all build backends. Cross compiling supports needs to be proposed and standardized for this sort of thing to work as command line flags, and I don't think we are close to that.

Another option, avoiding cross-compiling variables, would be to ask sysconfig, since _PYTHON_SYSCONFIGDATA_NAME is set to get sysconfig to report cross-compiling information. That's even better, really, since it's a Python specific thing and not related to meson itself. (Thought that depends on being able to generate the cross compile file in meson-python rather than picking up one)

A third option, which I'd like to avoid, but is already sadly being done for cmake, is to wrap the meson executable and inject the cross compiling command line flags whenever it is called. Environment variables are better than wrapping executables and injecting command line arguments.

Using backend-flags in meta.yaml to pass the meson cross-file seems to work fine (and is better than the patch):

backend-flags: setup-args=--cross-file=$PYODIDE_ROOT/tools/emscripten.meson.cross

The problem is cibuildwheel --platform pyodide (or emscripten, I forget was was chosen) should work without having to add custom configuration for meson projects. cibuildwheel can set environment variables (MESON_PYTHON_CROSSCOMPILE, for example), but it can't add command line flags that are not supported by all build backends.

Thanks, I understand the use case better now. I do think it can be solved fairly easily, since you're not actually flying blind in cibuildwheel. You can check pyproject.toml for build-backend = 'mesonpy', which is the only way to use meson-python, and only add the CLI flags if you know that meson-python is being used.

In general I think doing that will be helpful; the amount of standardized behavior is always going to be limited and lag way behind the actual state of the tools. It may also allow you to get rid of at least some of the undesirable cmake wrapping, for idiomatic use of scikit-build-core as a build backend.

You absolutely should not parse build-backend from pyproject.toml and customize behavior based on that. That would lock the feature to meson-python - you would eliminate the ability to fork meson-python or rename it or make a new meson-based backend, and you eliminate the ability to wrap the backend with a local addition (as mentioned by PEP 517, and used on some projects with scikit-build and setuptools!) - in fact, it would just mysteriously break without indicating why when you do something you are supposed to be able to do. One example: since meson calls cmake in some cases, you might wrap the backend to add the download of cmake if not present feature.

I'm willing to add a meson-python specific environment variable to cibuildwheel, I am not willing to parse the pyproject.toml and try to customize behavior based on assumed backends. (As in, I'm a -1, other cibuildwheel devs would weigh in too if it's proposed)

I think the best case would be for meson-python to check sysconfig, and if it's set to cross compile to emscripten and there's no cross-compile file already specified, it should add the cross-compile file. This doesn't require a special environment variable added, and is similar to the already agreed upon and added support for other cross compiles, like macOS ARM.

And second best would be for an meson-python specific environment variable that we could add to cibuildwheel.

Remaining things that needs to be looked at:

• Removing space after -L is needed in the CI (this is not only a local problem as I originally thought) see build log. As I mentioned above, -L gets separated from its argument (/root/repo/cpython/installs/python-3.11.3/lib/ in the CI case). A quick work-around would be to remove the space after -L in Makefile.envs. Otherwise any suggestions about how to track this down, more than welcome!
  Here is the LDFLAGS value:

LDFLAGS: -O2 -g0 -s MODULARIZE=1 -s LZ4=1 -L /root/repo/cpython/installs/python-3.11.3/lib/ -s WASM_BIGINT -s SIDE_MODULE=1

Somehow later down the line the /root/repo/cpython/installs/python-3.11.3/lib/ gets separated from -L in the emcc command:

[29/168] emcc  -o skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so     
skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so.p/meson-generated_fast
_exp.c.o -L -I/root/repo/cpython/installs/python-3.11.3/include/python3.11      
-sERROR_ON_UNDEFINED_SYMBOLS=0 -Wl,-O1 -shared -fPIC -O2 -g0 -s MODULARIZE=1 -s 
LZ4=1 /root/repo/cpython/installs/python-3.11.3/lib/ -s WASM_BIGINT -s          
SIDE_MODULE=1 -O2 -g0 -fPIC

which fails:

FAILED: skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so               
emcc  -o skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so              
skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so.p/meson-generated_fast
_exp.c.o -L -I/root/repo/cpython/installs/python-3.11.3/include/python3.11      
-sERROR_ON_UNDEFINED_SYMBOLS=0 -Wl,-O1 -shared -fPIC -O2 -g0 -s MODULARIZE=1 -s 
LZ4=1 /root/repo/cpython/installs/python-3.11.3/lib/ -s WASM_BIGINT -s          
SIDE_MODULE=1 -O2 -g0 -fPIC                                                     
wasm-ld: error: cannot open /root/repo/cpython/installs/python-3.11.3/lib/: Is a
directory                                                                       
emcc: error: '/root/repo/emsdk/emsdk/upstream/bin/wasm-ld -o                    
skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so --whole-archive       
skimage/_shared/fast_exp.cpython-311-wasm32-emscripten.so.p/meson-generated_fast
_exp.c.o -L-I/root/repo/cpython/installs/python-3.11.3/include/python3.11 -O1   
/root/repo/cpython/installs/python-3.11.3/lib/                                  
-L/root/repo/emsdk/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten
/pic --no-whole-archive -mllvm -combiner-global-alias-analysis=false -mllvm     
-enable-emscripten-sjlj -mllvm -disable-lsr --import-undefined --import-memory  
--strip-debug --export-dynamic --export-if-defined=main                         
--export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm            
--export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps  
--export-if-defined=__start_em_js --export-if-defined=__stop_em_js              
--export-if-defined=__main_argc_argv                                            
--export-if-defined=__wasm_apply_data_relocs                                    
--export-if-defined=__wasm_call_ctors --experimental-pic -shared' failed        
(returned 1)                                                                   

• Using exe_wrapper = 'node' yields an error ../../meson.build:1:0: ERROR: Executables created by c compiler emcc are not runnable. not sure whether we need this in the first place. I think this happens because by tweaking the LDFLAGS we produce a side module for the test program and not a .js file. Suggestions on how to change the LDFLAGS only for the scikit-image compilation more than welcome!

You absolutely should not parse build-backend from pyproject.toml and customize behavior based on that.

Agreed, also -1 for this.

I have cleaned things up a bit. I believe the CI errors are not related to this PR. Any feed-back let me know!

I'll go ahead and merge. I'm also seeing the same unrelated test failure on other PRs, though main is green. I don't know if it's some CI caching issue.