Skip to content

Bump to v0.18.6, finish py314 migration#114

Merged
sdvillal merged 8 commits intoconda-forge:mainfrom
sdvillal:finish-py314-migration
Feb 14, 2026
Merged

Bump to v0.18.6, finish py314 migration#114
sdvillal merged 8 commits intoconda-forge:mainfrom
sdvillal:finish-py314-migration

Conversation

@sdvillal
Copy link
Copy Markdown
Contributor

@sdvillal sdvillal commented Feb 2, 2026

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

This finishes the python 3.14 migration. #113 was never reopened by the autotick bot, I suspect that it got confused by us cherry-picking the commit but later reverting in #112.

Also removes "5.0" from arch list, as per @weiji14 comment in #112.

@conda-forge-admin
Copy link
Copy Markdown
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

@sdvillal sdvillal mentioned this pull request Feb 2, 2026
@sdvillal
Copy link
Copy Markdown
Contributor Author

sdvillal commented Feb 2, 2026

Seems that we are hitting this upstream bug, only on linux-64 + python 3.14:
deepspeedai/DeepSpeed#7673

Example traceback:

Traceback (most recent call last):
File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/engine_v2.py", line 18, in <module>
   from .model_implementations import InferenceV2Policy
 File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/model_implementations/__init__.py", line 12, in <module>
   from .llama_v2 import *
 File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/model_implementations/llama_v2/__init__.py", line 6, in <module>
   from .policy import Llama2Policy
 File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/model_implementations/llama_v2/policy.py", line 10, in <module>
   from .container import Llama2NonTransformerContainer, Llama2TransformerContainer
 File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/model_implementations/llama_v2/container.py", line 42, in <module>
   class Llama2TransformerContainer(LayerContainer):
   ...<20 lines>...
       }
 File "$PREFIX/lib/python3.14/site-packages/deepspeed/inference/v2/model_implementations/layer_container_base.py", line 97, in __new__
   raise ValueError(
       "Target parameter \"{}\" not found in this layer. Valid targets are {}".format(
           base_dependency, all_names))
ValueError: Target parameter "qkv_w" not found in this layer. Valid targets are []

@sdvillal
Copy link
Copy Markdown
Contributor Author

sdvillal commented Feb 2, 2026

We could simply disable tests in py3.14 for the time being, or try to lend a hand fixing upstream, or both. What do you think @weiji14?

@sdvillal sdvillal marked this pull request as draft February 2, 2026 08:30
@weiji14
Copy link
Copy Markdown
Member

weiji14 commented Feb 2, 2026

Maybe give the deepspeed devs a few days and see if they do a bugfix release. If not, we can skip just the failing test (don't disable tests entirely for py314), assuming other import namespaces are not affected.

Edit: actually, the test is failing on the top-level import deepspeed call, so we do need to wait for upstream...

@sdvillal
Copy link
Copy Markdown
Contributor Author

sdvillal commented Feb 5, 2026

Waiting for deepspeedai/DeepSpeed#7831

tohtana added a commit to deepspeedai/DeepSpeed that referenced this pull request Feb 10, 2026
Current metaclasses for layers and parameters access annotations in a
way that is incompatible with python 3.14+

See:
- [Python 3.14 release
notes](https://docs.python.org/3/whatsnew/3.14.html)
- [Porting
annotations](https://docs.python.org/3/whatsnew/3.14.html#whatsnew314-porting-annotations)
- [PEP649](https://peps.python.org/pep-0649/) and
[PEP749](https://peps.python.org/pep-0749/)

This PR uses annotationlib from python 3.14 onwards and keeps backwards
compatibility.

closes #7673
should unblock CF builds for py3.14
conda-forge/deepspeed-feedstock#114

A question is, does deepspeed support officially 3.14 yet? Should we
test it in CIs?

---------

Signed-off-by: Santi Villalba <sdvillal@gmail.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
@sdvillal
Copy link
Copy Markdown
Contributor Author

Upstream PR was merged, let's wait for a new release to be cut and update the PR.

@sdvillal sdvillal changed the title Finish py314 migration Bump to v0.18.6, finish py314 migration Feb 13, 2026
@sdvillal sdvillal force-pushed the finish-py314-migration branch from 45cb83a to 8969a03 Compare February 13, 2026 00:57
@sdvillal
Copy link
Copy Markdown
Contributor Author

Weirdly only in python 3.11 we are failing to find this header file:

In file included from csrc/aio/py_lib/deepspeed_py_aio.h:12,
                    from csrc/aio/py_lib/deepspeed_aio_op_desc.h:10:
    $PREFIX/include/torch/extension.h:6:10: fatal error: torch/all.h: No such file or directory
        6 | #include <torch/all.h>
           |          ^~~~~~~~~~~~~
    compilation terminated.

Problem with the pytorch package?

@sdvillal
Copy link
Copy Markdown
Contributor Author

sdvillal commented Feb 13, 2026

Here the weird difference between the failing python 3.11 build and the successful python 3.12:

# GOOD: $BUILD_PREFIX/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/deepspeed-0.18.6 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -I$PREFIX/targets/x86_64-linux/include -I$BUILD_PREFIX/targets/x86_64-linux/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -I$PREFIX/targets/x86_64-linux/include -I$BUILD_PREFIX/targets/x86_64-linux/include -fPIC -I$SRC_DIR/csrc/aio/py_lib -I$SRC_DIR/csrc/aio/common -I$PREFIX/include -I$PREFIX/lib/python3.12/site-packages/torch/include -I$PREFIX/include/torch/csrc/api/include -I$PREFIX/lib/python3.12/site-packages/torch/include/torch/csrc/api/include -I$PREFIX/include/python3.12 -c csrc/aio/py_lib/deepspeed_aio_op_desc.cpp -o build/temp.linux-x86_64-cpython-312/csrc/aio/py_lib/deepspeed_aio_op_desc.o -O3 -g -Wno-reorder -march=native -fopenmp -D__AVX256__ -Wall -O0 -shared -fPIC -Wno-reorder -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=async_io_op -std=c++17
# https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1461372&view=logs&j=5eaeabf9-98bb-5ad4-ae9b-9c34ad8b134f&t=f9feff57-a1a8-511d-d6e0-bac292e08e21&l=3963

# BAD: $BUILD_PREFIX/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/deepspeed-0.18.6 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -I$PREFIX/targets/x86_64-linux/include -I$BUILD_PREFIX/targets/x86_64-linux/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -I$PREFIX/targets/x86_64-linux/include -I$BUILD_PREFIX/targets/x86_64-linux/include -fPIC -I$SRC_DIR/csrc/aio/py_lib -I$SRC_DIR/csrc/aio/common -I/home/conda/feedstock_root/build_artifacts/python-split_1769470388965/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -I$PREFIX/lib/python3.11/site-packages/torch/include -I/home/conda/feedstock_root/build_artifacts/python-split_1769470388965/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/torch/csrc/api/include -I$PREFIX/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I$PREFIX/include/python3.11 -c csrc/aio/py_lib/deepspeed_aio_op_desc.cpp -o build/temp.linux-x86_64-cpython-311/csrc/aio/py_lib/deepspeed_aio_op_desc.o -O3 -g -Wno-reorder -march=native -fopenmp -D__AVX256__ -Wall -O0 -shared -fPIC -Wno-reorder -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=async_io_op -std=c++17
# https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1461372&view=logs&j=d49dbcef-bf20-5a8a-a3c0-6b6309067d07&t=e281c35a-03d0-5d77-85e5-21b13dd3ac34&l=3963

Which I feel boils down to the weird include line in python 3.11:
-I/home/conda/feedstock_root/build_artifacts/python-split_1769470388965/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -I$PREFIX/lib/python3.11/site-packages/torch/include -I/home/conda/feedstock_root/build_artifacts/python-split_1769470388965/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/torch/csrc/api/include

Do you have a quick idea of what might be happening here with the python 3.11 build @h-vetinari @mgorny ?

@mgorny
Copy link
Copy Markdown

mgorny commented Feb 13, 2026

Quite weird. There were some header-related changes recently, but it seems that both builds used a matching pytorch builds, and the same libtorch version. I'll try to reproduce it locally.

@mgorny
Copy link
Copy Markdown

mgorny commented Feb 13, 2026

Ok, clearly something went wrong and $PREFIX substitution doesn't happen in the py3.11 package. The real question would be why, given they're all being built in a single job.

@mgorny
Copy link
Copy Markdown

mgorny commented Feb 13, 2026

Wait a minute, that path is coming from Python itself!

Python 3.11.14 | packaged by conda-forge | (main, Jan 26 2026, 23:48:32) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("prefix")
'/home/conda/feedstock_root/build_artifacts/python-split_1769470388965/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol'

Curious enough, I can't reproduce it in plain conda env:

Python 3.11.14 | packaged by conda-forge | (main, Jan 26 2026, 23:48:32) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("prefix")
'/srv/p2p/conda/miniforge3/envs/test'

Is it a bug in rattler-build perhaps?

@mgorny
Copy link
Copy Markdown

mgorny commented Feb 13, 2026

Filed prefix-dev/rattler-build#2147

@sdvillal
Copy link
Copy Markdown
Contributor Author

Thanks a lot for looking @mgorny !

waiting for prefix-dev/rattler-build#2147 to be resolved
will open a PR for the opposite selection, and remove the selector on a ci skip later
@sdvillal
Copy link
Copy Markdown
Contributor Author

Pragmatically, we are now skipping all python 3.11 builds - likely unnecessarily broad but also minimal impact as we have a py3.11 package for the recent previous patch version.

The key is to already enable people who need a python 3.14 build with an option.

Plan: I will open a new PR for python 3.11. Let's keep an eye on the rattler-build fix for the prefix expansion problem, merge when ready, and finally remove the python version selector directly in main with a ci skip commit.

@weiji14 FYI

@sdvillal sdvillal marked this pull request as ready for review February 14, 2026 08:37
@mgorny
Copy link
Copy Markdown

mgorny commented Feb 14, 2026

I haven't tested it, but I think a cheap workaround would be to find ${SP_DIR} -name '*.pyc' -delete.

@sdvillal sdvillal merged commit 084e746 into conda-forge:main Feb 14, 2026
19 checks passed
This was referenced Feb 14, 2026
@sdvillal
Copy link
Copy Markdown
Contributor Author

I haven't tested it, but I think a cheap workaround would be to find ${SP_DIR} -name '*.pyc' -delete.

I tried (hopefully correctly) but it seems it did not work
411660f

nathon-lee pushed a commit to nathon-lee/DeepSpeed_woo that referenced this pull request Mar 7, 2026
Current metaclasses for layers and parameters access annotations in a
way that is incompatible with python 3.14+

See:
- [Python 3.14 release
notes](https://docs.python.org/3/whatsnew/3.14.html)
- [Porting
annotations](https://docs.python.org/3/whatsnew/3.14.html#whatsnew314-porting-annotations)
- [PEP649](https://peps.python.org/pep-0649/) and
[PEP749](https://peps.python.org/pep-0749/)

This PR uses annotationlib from python 3.14 onwards and keeps backwards
compatibility.

closes deepspeedai#7673
should unblock CF builds for py3.14
conda-forge/deepspeed-feedstock#114

A question is, does deepspeed support officially 3.14 yet? Should we
test it in CIs?

---------

Signed-off-by: Santi Villalba <sdvillal@gmail.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
nathon-lee pushed a commit to nathon-lee/DeepSpeed_woo that referenced this pull request Mar 28, 2026
Current metaclasses for layers and parameters access annotations in a
way that is incompatible with python 3.14+

See:
- [Python 3.14 release
notes](https://docs.python.org/3/whatsnew/3.14.html)
- [Porting
annotations](https://docs.python.org/3/whatsnew/3.14.html#whatsnew314-porting-annotations)
- [PEP649](https://peps.python.org/pep-0649/) and
[PEP749](https://peps.python.org/pep-0749/)

This PR uses annotationlib from python 3.14 onwards and keeps backwards
compatibility.

closes deepspeedai#7673
should unblock CF builds for py3.14
conda-forge/deepspeed-feedstock#114

A question is, does deepspeed support officially 3.14 yet? Should we
test it in CIs?

---------

Signed-off-by: Santi Villalba <sdvillal@gmail.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants