Skip to content

Add test-simd package to validate WASM/SSE/AVX intrinsics #5880

Merged
ryanking13 merged 12 commits intopyodide:mainfrom
teddygood:feat/issue-5855-basic-simd-tests
Sep 16, 2025
Merged

Add test-simd package to validate WASM/SSE/AVX intrinsics #5880
ryanking13 merged 12 commits intopyodide:mainfrom
teddygood:feat/issue-5855-basic-simd-tests

Conversation

@teddygood
Copy link
Copy Markdown
Contributor

@teddygood teddygood commented Sep 8, 2025

Description

resolve #5855

  • Implemented minimal shared libraries with intrinsics for:
    • WASM SIMD (wasm_simd128.h, f32x4 add/mul)
    • SSE1 (_mm_add_ps, _mm_mul_ps, f32x4)
    • SSE2 (_mm_add_pd, _mm_mul_pd, f64x2)
    • AVX (_mm256_add_ps, _mm256_mul_ps, f32x8, using duplicated 128b halves)
  • Added pytest integration to dynamically load the .so libraries via pyodide.loadPackage("test-simd") and compare scalar results against expected values.

For the AVX case, WebAssembly SIMD only works with 128-bit vectors.
Emscripten converts each 256-bit AVX instruction into two 128-bit ones.
In our test, we copied the lower 128 bits into the upper half to fill the 256-bit register.
Because of this, the scalar sum equals twice the SSE result. This shows that the upper 128 bits are used and the lowering works as expected.

Checklist

  • Add / update tests

Copy link
Copy Markdown
Member

@ryanking13 ryanking13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @teddygood! Overall it looks good, but I left some tips for creating a test package for this.

Comment on lines +7 to +12
- library
- shared_library
source:
path: src
build:
type: shared_library
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of making a shared_library, let's write a Python package that imports the shared library. It is much easy to test and read, while it adds a little bit of more code.

You can refer to the dummy-nonpure package to see how you can write a Python module that is written in C.

  1. I think you can keep the existing implementation, while making a wrapper Python function.

For example, the Python function can be defined such as:

// Python wrapper for simd_avx_add8_sum
static PyObject *
simd_avx_add8_sum(PyObject *self, PyObject *args)
{
  float a0, a1, a2, a3, b0, b1, b2, b3;

  if (!PyArg_ParseTuple(args, "ffffffff", &a0, &a1, &a2, &a3, &b0, &b1, &b2, &b3))
    return NULL;

  float result = simd_avx_add8_sum_impl(a0, a1, a2, a3, b0, b1, b2, b3);

  return Py_BuildValue("f", result);
}
  1. The build script can also be moved to setup.py, such as
import os
from setuptools import setup, Extension

module = Extension(
    'test_simd.simd_wrapper',
    sources=[os.path.join('test_simd', 'simd_avx.c')],
    extra_compile_args=['-mavx', '-msimd128'],
)
  1. Then it can be tested such as
@run_in_pyodide(packages=["test-simd"])
def test_simd_functions(selenium):
    from test_simd import simd_wrapper
    a = (1.0, 2.0, 3.0, 4.0)
    b = (5.0, 6.0, 7.0, 8.0)

    simd_wrapper.avx_mul(*a, *b)

(The code in this comment is not verified, so some details might be incorrect)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review! I’ll make the changes and commit them.

Comment on lines +10 to +12
build:
type: cpython_module
script: python -m pip install . -v
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
build:
type: cpython_module
script: python -m pip install . -v

These are not needed.

  • cpython_module type is for unvendored CPython stdlibs
  • build script will automatically run for normal Python packages.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I’ll update it.

@teddygood
Copy link
Copy Markdown
Contributor Author

teddygood commented Sep 14, 2025

1. In the macOS (arm64) build, test-simd fails because it includes x86-only SIMD headers such as <immintrin.h> and <mmintrin.h>. These headers provide SSE, AVX, and MMX intrinsics that are not supported on arm64, causing Clang to raise errors. Therefore, the SIMD test code needs to be conditionally excluded or adapted for non-x86 platforms.

2. The Ubuntu build failed because the -msimd128 flag, which is only valid for WebAssembly with Emscripten/Clang, was passed to GCC, which does not support it. The command also mixed x86 flags (-msse2, -mavx) with WebAssembly-specific flags, causing a conflict.

I will resolve the above issue and test it again.

  1. Always read the documentation carefully.
  2. Always read the logs thoroughly.

I learned something again today. Thank you...

Comment on lines +4 to +5
tag:
- pyodide.test
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tag:
- pyodide.test
tag:
- core
- pyodide.test

Could you add tag: core as well? It seems like this package is not tested in CI because of a missing tag.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I’ll add it right away.

Copy link
Copy Markdown
Member

@ryanking13 ryanking13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ryanking13 ryanking13 merged commit 7cd3d8a into pyodide:main Sep 16, 2025
37 of 38 checks passed
Drranny pushed a commit to Drranny/pyodide that referenced this pull request Feb 15, 2026
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Experiment with basic SIMD operations

2 participants