@ngoldbaum perhaps you would be interested in making a TSan version based on what I have here? I suppose to test that this is understandable and maintainable.

@lucascolley, could we extend this to linux-x86_64 too? I feel like if we are gonna upstream the CPython recipes, we would want at least these two platforms. I can help with that if needed 😊

yes, go for it! I only have an osx-arm64 machine right now (although not really an excuse for linux with docker...) but will have access to a linux and a windows box next week. Feel free to beat me to it ☺️

perhaps you would be interested in making a TSan version based on what I have here? I suppose to test that this is understandable and maintainable.

Sure, is there a CPython recipe I should be using? How do I use this recipe with the CPython recipe?

Maybe it'd help to have a README in the new pixi_packages folder that explains to luddites like me how to use the recipes to do useful development tasks.

I'm figuring out the details for CPython now, will let you know when ready.

This will need a release note.

@ngoldbaum

Sure, is there a CPython recipe I should be using? How do I use this recipe with the CPython recipe?

I pushed osx-arm64 recipes for default and ASan CPython at lucascolley/cpython@375594b. I have updated this PR so that the NumPy ASan package builds against the CPython ASan package.

Hopefully it is clear how one would go about adding another build type like TSan:

• add linux support to the CPython recipes if needed, by adding "linux-64" to workspace.platforms in the pixi.toml files, seeing what fails on pixi build, and making the necessary adjustments
• add support for the build type in the CPython build script, https://github.com/lucascolley/cpython/blob/375594b556fb6c41c0630c10b7e934fb89604712/Tools/pixi-packages/build.sh
• add a new directory with pixi.toml and recipe.yaml for that build type in CPython, like https://github.com/lucascolley/cpython/blob/375594b556fb6c41c0630c10b7e934fb89604712/Tools/pixi-packages/asan/recipe.yaml
• add a new directory and pixi.toml for that build type in NumPy and point to the appropriate CPython directory, like https://github.com/lucascolley/numpy/blob/9f05918520d2579134758b63a45da9a7ebcacaa3/pixi-packages/asan/pixi.toml
• those packages can then be added as dependencies in a Pixi workspace, as in BLD/DEV/CI: ASAN at your fingertips! scipy/scipy#24066.

Maybe it'd help to have a README in the new pixi_packages folder that explains to luddites like me how to use the recipes to do useful development tasks.

Development tasks on which repo? For NumPy, we should probably wait until something like https://github.com/scipy/scipy/blob/main/pixi.toml exists, but they could also be added to an out-of-tree workspace like https://github.com/rgommers/pixi-dev-scipystack/blob/main/numpy/pixi.toml.

For SciPy, we should add some docs, yes, and it would probably be sufficient for a README here to link to those docs?

For SciPy, we should add some docs, yes, and it would probably be sufficient for a README here to link to those docs?

Sounds reasonable to me.

Thanks so much for working on this. I was hoping you would when you shared all this stuff earlier this year when it was beta.

my pleasure — it has been super cool to work across both the development and user side of this!

@FFY00 what information would be good to include on the CPython issue?

For NumPy, we should probably wait until something like https://github.com/scipy/scipy/blob/main/pixi.toml exists

gh-30387

@FFY00 I went ahead and added linux-64 support at lucascolley/cpython@54274f2, lucascolley@aa50cf4, and scipy/scipy@1d87a86. Tests now running at https://github.com/scipy/scipy/actions/runs/20012600510/job/57384436353.

Sorry for the delay getting back to you.

Thank you so much for all this work, it is extremely helpful!

@FFY00 what information would be good to include on the CPython issue?

I opened an issue at python/cpython#142466. Feel free to send a PR whenever you are ready!

The CPython PR looks like will be ready soon, I'll give it one week-ish to give time for others to object, but if nobody does, I'll merge it 😃

🎉🎉

Would you have time to address Hugo's comment at python/cpython#142469 (comment) @FFY00 ? I am running out of the ability to justify procrastinating my spare time away on this to myself 😅 .

Done. Could you just confirm the llvm-tools thing on macos, since IIRC you use that platform? Other than that, we are good 👍

replied at python/cpython#142469 (comment)

All good in CPython, I'll probably merge it on the weekend, or start of next week. Congrats @lucascolley!

Seems like we are happy punting Windows down the line

Yeah that's fine I've never tried to do this exercise with sanitizers on Windows and am not aware of anyone else doing that

I added a README and a simple CI job which invokes pixi build for each platform and variant.

I also added a release note.

CI failures look unrelated

I'm testing asan on my Linux box:

[workspace]
channels = ["conda-forge"]
name = "cpython-pixi-test"
platforms = ["linux-64"]
preview = ["pixi-build"]

[dependencies]
python = { git = "https://github.com/python/cpython", subdirectory = "Tools/pixi-packages/asan" }
numpy = { git = "https://github.com/lucascolley/numpy", rev = "pixi-build", subdirectory = "pixi-packages/asan" }

It works, but it's extremely verbose, producing several hundreds of pages of output. Unsure if they are legit issues with numpy or just noise? Or am I using it incorrectly?

$ pixi s
$ python -c 'import numpy'

[... too many lines of  output to copy-paste. here is just one ...]
Indirect leak of 14 byte(s) in 1 object(s) allocated from:
    #0 0x782aed6fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x782ae833e896 in PyArrayMethod_FromSpec_int ../numpy/_core/src/multiarray/array_method.c:481
    #2 0x782ae8594f9b in PyUFunc_AddLoopFromSpec_int ../numpy/_core/src/umath/dispatching.cpp:177
    #3 0x782ae86592e8 in init_ufunc(_object*, char const*, PyArray_DTypeMeta**, NPY_CASTING (*)(PyArrayMethodObject_tag*, PyArray_DTypeMeta* const*, _PyArray_Descr* const*, _PyArray_Descr**, long*), int (*)(PyArrayMethod_Context_tag*, char* const*, long const*, long const*, NpyAuxData_tag*), int, int, NPY_CASTING, NPY_ARRAYMETHOD_FLAGS, void*) ../numpy/_core/src/umath/stringdtype_ufuncs.cpp:2492

SUMMARY: AddressSanitizer: 35505 byte(s) leaked in 410 allocation(s).

do you have something like https://github.com/lucascolley/scipy/blob/78445c90e2dc2560d8f6ae044fab0d6d0eaa08ea/pixi.toml#L598

do you have something like https://github.com/lucascolley/scipy/blob/78445c90e2dc2560d8f6ae044fab0d6d0eaa08ea/pixi.toml#L598

Ah yes, that quiets it down.

Hmmm, fun CI failures. The MacOS ones are spurious matmul runtime warnings, the windows ones looked like new warnings generated by cython code. Retriggered, but just out of curiosity...

Sorry, nvm. would need a re-base for the new allowed warnings. And that is just not worth it.