Skip to content

Build Python 3.13 wheels (not free-threaded)#711

Merged
jcrist merged 15 commits into
msgspec:mainfrom
edgarrmondragon:py313
Oct 13, 2024
Merged

Build Python 3.13 wheels (not free-threaded)#711
jcrist merged 15 commits into
msgspec:mainfrom
edgarrmondragon:py313

Conversation

@edgarrmondragon

@edgarrmondragon edgarrmondragon commented Jul 13, 2024

Copy link
Copy Markdown
Contributor

I can close this if #703 gets more attention, but I've confirmed changes here allow the wheels to be built in 3.8-3.13, and all tests are passing.

This warning is currently raised, but fixing it might be beyond the scope of this PR:

PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None

python/cpython#106004

@edgarrmondragon edgarrmondragon left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is ready for an initial review.

Mostly followed advice from

Comment thread .github/workflows/ci.yml Outdated
Comment thread msgspec/_utils.py Outdated
value = type(None)
elif isinstance(value, str):
value = _forward_ref(value)
# TODO: Pass `type_params` to `_eval_type` to fix deprecation warning

@edgarrmondragon edgarrmondragon Jul 13, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This missing parameter is causing a ton of warnings like this:

msgspec/_utils.py:130: DeprecationWarning: Failing to pass a value to the 'type_params' parameter of 'typing._eval_type' is deprecated, as it leads to incorrect behaviour when calling typing._eval_type on a stringified annotation that references a PEP 695 type parameter. It will be disallowed in Python 3.15.

python/cpython#118418

@edgarrmondragon edgarrmondragon marked this pull request as ready for review July 13, 2024 07:08
@edgarrmondragon

edgarrmondragon commented Jul 13, 2024

Copy link
Copy Markdown
Contributor Author

This warning is currently raised but fixing it might be beyond the scope of this PR to fix it:

PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None

FWIW I think this is hiding an actual error:

PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None
    
    Traceback (most recent call last):
      File "/opt/_internal/cpython-3.13.0b3/lib/python3.13/typing.py", line 2251, in __hash__
        return hash((self.__origin__, self.__metadata__))
    TypeError: unhashable type: 'set'
    
      warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

in

https://github.com/jcrist/msgspec/blob/2c37da090b2c5fcb2ecca9ae00274c67fabb85cf/tests/test_inspect.py#L790-L793

Comment thread .github/workflows/ci.yml Outdated
@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

macos-11 runners aren't available anymore, so I updated it to macos-14 in 748f4bb.

https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/#macos-11-deprecation-and-removal

@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

macos-11 runners aren't available anymore, so I updated it to macos-14 in 748f4bb.

https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/#macos-11-deprecation-and-removal

With that change, the wheels are finally built instead of the workflow getting stuck forever:

 12 wheels produced in 5 minutes:
  msgspec-0+untagged.1.g766a8f7-cp310-cp310-macosx_10_9_x86_64.whl    186 kB
  msgspec-0+untagged.1.g766a8f7-cp310-cp310-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp311-cp311-macosx_10_9_x86_64.whl    184 kB
  msgspec-0+untagged.1.g766a8f7-cp311-cp311-macosx_11_0_arm64.whl     178 kB
  msgspec-0+untagged.1.g766a8f7-cp312-cp312-macosx_10_9_x86_64.whl    186 kB
  msgspec-0+untagged.1.g766a8f7-cp312-cp312-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp313-cp313-macosx_10_13_x86_64.whl   186 kB
  msgspec-0+untagged.1.g766a8f7-cp313-cp313-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp38-cp38-macosx_10_9_x86_64.whl      186 kB
  msgspec-0+untagged.1.g766a8f7-cp38-cp38-macosx_11_0_arm64.whl       180 kB
  msgspec-0+untagged.1.g766a8f7-cp39-cp39-macosx_10_9_x86_64.whl      186 kB
  msgspec-0+untagged.1.g766a8f7-cp39-cp39-macosx_11_0_arm64.whl       180 kB

https://github.com/edgarrmondragon/msgspec/actions/runs/10030128227/job/27719084757

@tacaswell

Copy link
Copy Markdown
Contributor

I can also get msgspec to build with py313 with this branch, but not the branch in #703

@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

@jcrist This is ready for review at your convenience :)

@edgarrmondragon edgarrmondragon mentioned this pull request Aug 6, 2024
@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

Updated to use Python 3.13.0rc1

@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

Updated to use Python 3.13.0rc2

@ofek

ofek commented Sep 18, 2024

Copy link
Copy Markdown
Member

@jcrist Can you please run the CI?

@Tishka17

Tishka17 commented Oct 2, 2024

Copy link
Copy Markdown

Python 3.13 release is scheduled for then upcoming Monday. Are there any plans to merge this PR and publish a new version?

@Tishka17

Tishka17 commented Oct 7, 2024

Copy link
Copy Markdown

Python 3.13 is out there.

@maurerle

maurerle commented Oct 11, 2024

Copy link
Copy Markdown

This closes #741 and #698

I was building on debian 13 trixie which uses GCC 14, I therefore had a few more required adjustments on top of yours:

maurerle@2aa5db5

To build this locally, I used conda:

conda create -n py313 python-freethreading -c conda-forge
conda activate pyt313
git clone -b py313 git@github.com:maurerle/msgspec.git
cd msgspec
pip install .

@maurerle

maurerle commented Oct 11, 2024

Copy link
Copy Markdown

I do get a segfault though when importing msgspec.. Even with GIL enabled

(py313) ➜  msgspec git:(py313) ✗ python -X gil=1 -X faulthandler
Python 3.13.0 experimental free-threading build | packaged by conda-forge | (main, Oct  8 2024, 20:16:19) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import msgspec
Fatal Python error: Segmentation fault

Current thread 0x00007fcd5e8ba640 (most recent call first):
  File "~/github/msgspec/msgspec/__init__.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<python-input-1>", line 1 in <module>

@edgarrmondragon

Copy link
Copy Markdown
Contributor Author

I do get a segfault though when importing msgspec.. Even with GIL enabled

(py313) ➜  msgspec git:(py313) ✗ python -X gil=1 -X faulthandler
Python 3.13.0 experimental free-threading build | packaged by conda-forge | (main, Oct  8 2024, 20:16:19) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import msgspec
Fatal Python error: Segmentation fault

Current thread 0x00007fcd5e8ba640 (most recent call first):
  File "~/github/msgspec/msgspec/__init__.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<python-input-1>", line 1 in <module>

@maurerle thanks for finding that. I'll give a read to https://py-free-threading.github.io/porting/, unless someone beats me to it and knows what's wrong here.

@maurerle

maurerle commented Oct 12, 2024

Copy link
Copy Markdown

When executing this on a gil build (e.g. conda create -n py313-gil python=3.13)
One gets an error if executed from the build directory:

>>> import msgspec
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import msgspec
  File "~/github/msgspec/msgspec/__init__.py", line 1, in <module>
    from ._core import (
    ...<14 lines>...
    )
ModuleNotFoundError: No module named 'msgspec._core'

So make sure to execute this not in the build directory :)
It works fine then :)
But I don't have a clue how to solve the above segfault either.

The segfault also occurs when starting the above mentioned freethreading build with an active GIL as in
python -X gil=1 -c "import msgspec"

@jcrist jcrist merged commit 7ade469 into msgspec:main Oct 13, 2024
@jcrist

jcrist commented Oct 13, 2024

Copy link
Copy Markdown
Member

Thanks for the PR (and everyone for their patience). With a new kid and recent layoff, I've been a bit busy and msgspec maintenance has languished. I really appreciate the help and patience here.

I have a few small fixups I want to land with this release, but I'd expect a release supporting Python 3.13 (not with freethreading) out in the next day or two. I'll work on adding freethreading support after getting that initial release out.

@jcrist jcrist mentioned this pull request Oct 13, 2024
5 tasks
@edgarrmondragon edgarrmondragon deleted the py313 branch October 14, 2024 00:36
@ekneg54

ekneg54 commented Oct 19, 2024

Copy link
Copy Markdown

Hi,

with this merged I thought that I should be able to install msgspec on python 3.13. But it turns out, that this is not the case as you could see:
image

I wonder if I am missing something. Could you help?

@maurerle

Copy link
Copy Markdown

You are missing the python3-dev dependency.
Install it using your system package manager

@mj0nez

mj0nez commented Oct 19, 2024

Copy link
Copy Markdown

It was merged but not released or am I mistaken? An install from main should work for sure: install from vcs

@ekneg54

ekneg54 commented Oct 19, 2024

Copy link
Copy Markdown

thank you. I wait for the release. Do you know when it is planned?

@mj0nez

mj0nez commented Oct 19, 2024

Copy link
Copy Markdown

No, but soon I guess.

@st3fan

st3fan commented Dec 22, 2024

Copy link
Copy Markdown

@jcrist Is there anything the community can do to help to get a release out? There is probably some collective spare hacking time available during the break now. Let us know if you need help testing or writing release notes, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Doesn't install on python 3.13 3.13 support

9 participants