Skip to content

Conversation

@zshipko
Copy link
Contributor

@zshipko zshipko commented Nov 16, 2023

Support for extism/extism#578

@chrisdickinson chrisdickinson self-requested a review December 11, 2023 20:22
@chrisdickinson
Copy link

chrisdickinson commented Dec 11, 2023

I pushed a commit up for the broken CI tests, which fixed ubuntu and macOS, but that windows failure is surprising!

Edit: looking at the build that released extism_sys rc6, I don't see any smoking guns in the logs. I wonder if the function is, in fact, missing from the DLL? I'll take a look at the artifact shortly.

Edit 2: It looks like it's present in the shared object. Running wheel unpack extism_sys-1.0.0rc6-py3-none-win_amd64.whl; hexdump -C extism_sys-1.0.0rc6/extism_sys/native.so yields the following:

00d16a90  61 63 65 00 65 78 74 69  73 6d 5f 6c 6f 67 5f 63  |ace.extism_log_c|
00d16aa0  75 73 74 6f 6d 00 65 78  74 69 73 6d 5f 6c 6f 67  |ustom.extism_log|
00d16ab0  5f 64 72 61 69 6e 00 65  78 74 69 73 6d 5f 6c 6f  |_drain.extism_lo|
00d16ac0  67 5f 66 69 6c 65 00 65  78 74 69 73 6d 5f 70 6c  |g_file.extism_pl|
00d16ad0  75 67 69 6e 5f 63 61 6c  6c 00 65 78 74 69 73 6d  |ugin_call.extism|

(This is pretty unscientific, though – it might not be exported. I'm not too handy with introspecting the DLL format; if you know of any tools that run on ubuntu or linux please let me know!)

@nilslice
Copy link
Member

@chrisdickinson - something like this might help?

readelf -S -C -s /path/to/lib.so  | grep extism_

@chrisdickinson
Copy link

chrisdickinson commented Dec 11, 2023

A-ha, thanks, that set me on the right path: I used readpe to introspect the (windows) DLL on my (macOS) laptop. It claims that the function is, in fact, exported by the DLL (readpe -e native.so):

            Function
                Ordinal:                         11
                Address:                         0x1c880
                Name:                            extism_log_custom
            Function
                Ordinal:                         12
                Address:                         0x1ce90
                Name:                            extism_log_drain

(Unfortunately this only deepens the mystery. I'll have to spin up a windows VM to investigate further.)

@chrisdickinson
Copy link

OK, I think I might've narrowed this down: the shared object has the function, but the python wheel doesn't list it in its generated metadata. I have a theory: this looks like the only place we directly accept a function pointer as a function parameter, and MSVC might struggle to parse that type when Maturin goes to generate the wheel.

I'll test this out tomorrow with a dev build of extism/extism that creates a typedef for the function pointer.

chrisdickinson pushed a commit to extism/extism that referenced this pull request Dec 15, 2023
This fixes a test failure on the python-sdk [1]. (See also [2]). In particular,
while maturin creates bindings for `extism_log_drain` on clang platforms, it
seems that MSVC cannot generate the required binding information for `ffi.py`.
This results in an `extism_sys` wheel whose shared object (in this case, a DLL)
contains the `extism_log_drain`, but whose Python FFI bindings don't contain a
definition for that function.

Naming the function pointer type parameter resolves the issue.

[1]: https://github.com/extism/python-sdk/actions/runs/7172934060/job/19669775001#step:9:35
[2]: extism/python-sdk#18 (comment)
chrisdickinson pushed a commit to extism/extism that referenced this pull request Dec 15, 2023
This fixes a test failure on the python-sdk [1]. (See also [2]). In particular,
while maturin creates bindings for `extism_log_drain` on clang platforms, it
seems that MSVC cannot generate the required binding information for `ffi.py`.
This results in an `extism_sys` wheel whose shared object (in this case, a DLL)
contains the `extism_log_drain`, but whose Python FFI bindings don't contain a
definition for that function.

Naming the function pointer type parameter resolves the issue.

[1]: https://github.com/extism/python-sdk/actions/runs/7172934060/job/19669775001#step:9:35
[2]: extism/python-sdk#18 (comment)
chrisdickinson pushed a commit to extism/extism that referenced this pull request Dec 15, 2023
This fixes a test failure on the python-sdk [1]. (See also [2]). In
particular, while maturin creates bindings for `extism_log_drain` on
clang platforms, it seems that MSVC cannot generate the required binding
information for `ffi.py`. This results in an `extism_sys` wheel whose
shared object (in this case, a DLL) contains the `extism_log_drain`, but
whose Python FFI bindings don't contain a definition for that function.

Naming the function pointer type parameter resolves the issue. What a
strange issue!

[1]:
https://github.com/extism/python-sdk/actions/runs/7172934060/job/19669775001#step:9:35
[2]:
extism/python-sdk#18 (comment)
@chrisdickinson chrisdickinson merged commit 279be90 into main Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants