Expr is not fully picklable in Python 3.14 #12043
Description
Describe the issue:
Since dask/dask-expr#331, assert_eq in the tests has checked that the Expr is picklable. With Python 3.14, almost 90 or so tests fail with a PicklingError:
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise[<lambda>18] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise[<lambda>20] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise[<lambda>26] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise[<lambda>28] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_rename - _pickle.PicklingError: Can't pickle local object <function test_rename.<loc...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise_pandas_only_warning[<lambda>0] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise_pandas_only_warning[<lambda>1] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise_pandas_only_warning[<lambda>2] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_blockwise_pandas_only_warning[<lambda>3] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_map_meta - _pickle.PicklingError: Can't pickle local object <function test_map_meta.<l...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_parent_mixed_column_assignment - _pickle.PicklingError: Can't pickle local object <function test_parent_mixe...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_apply_infer_columns - _pickle.PicklingError: Can't pickle local object <function test_apply_infer...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_combine - _pickle.PicklingError: Can't pickle local object <function test_combine.<lo...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_column_projection_map_partitions[None] - _pickle.PicklingError: Can't pickle local object <function test_column_proj...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_column_projection_map_partitions[required_columns1] - _pickle.PicklingError: Can't pickle local object <function test_column_proj...
FAILED dask/dataframe/dask_expr/tests/test_collection.py::test_dot - _pickle.PicklingError: Can't pickle local object <function FrameBase.dot.<l...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_reduction_optimize - _pickle.PicklingError: Can't pickle local object <function test_groupby_red...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_group_keys[True] - _pickle.PicklingError: Can't pickle local object <function test_groupby_gro...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_group_keys[False] - _pickle.PicklingError: Can't pickle local object <function test_groupby_gro...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_group_keys[None] - _pickle.PicklingError: Can't pickle local object <function test_groupby_gro...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_agg[1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_agg...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_agg[1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_agg...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_agg[2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_agg...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_agg[2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_agg...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-True-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-True-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-True-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-True-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-False-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-False-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-False-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-1-False-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-True-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-True-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-True-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-True-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-False-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-False-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-False-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[2-None-False-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-True-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-True-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-True-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-True-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-False-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-False-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-False-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-1-False-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-True-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-True-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-True-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-True-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-False-1-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-False-1-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-False-2-1] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_apply[None-None-False-2-2] - _pickle.PicklingError: Can't pickle local object <function test_rolling_app...
FAILED dask/dataframe/dask_expr/tests/test_rolling.py::test_rolling_one_element_window_empty_after - _pickle.PicklingError: Can't pickle local object <function test_rolling_one...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_apply - _pickle.PicklingError: Can't pickle local object <function test_groupby_app...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_transform - _pickle.PicklingError: Can't pickle local object <function test_groupby_tra...
FAILED dask/dataframe/dask_expr/tests/test_reductions.py::test_reduction_method - _pickle.PicklingError: Can't pickle local object <function test_reduction_m...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_numeric_only_lambda_caller - _pickle.PicklingError: Can't pickle local object <function test_groupby_num...
FAILED dask/dataframe/dask_expr/tests/test_reductions.py::test_reduction_split_every_false - _pickle.PicklingError: Can't pickle local object <function test_reduction_s...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_apply_or_transform_shuffle_multilevel[<lambda>0-True] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_apply_or_transform_shuffle_multilevel[<lambda>0-False] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_apply_or_transform_shuffle_multilevel[<lambda>1-True] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_apply_or_transform_shuffle_multilevel[<lambda>1-False] - _pickle.PicklingError: Can't pickle local object <function <lambda>.<locals...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_numeric_column_names - _pickle.PicklingError: Can't pickle local object <function test_numeric_col...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_apply_divisions - _pickle.PicklingError: Can't pickle local object <function test_apply_divis...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_apply_args - _pickle.PicklingError: Can't pickle local object <function test_groupby_app...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_udf_user_warning - _pickle.PicklingError: Can't pickle local object <function test_groupby_udf...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_dataframe_groupby_agg_custom_sum - _pickle.PicklingError: Can't pickle local object <function test_dataframe_g...
FAILED dask/dataframe/dask_expr/tests/test_groupby.py::test_groupby_getitem_apply_group_keys - _pickle.PicklingError: Can't pickle local object <function test_groupby_get...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_partitions - _pickle.PicklingError: Can't pickle local object <function test_map_partiti...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_partitions_broadcast - _pickle.PicklingError: Can't pickle local object <function test_map_partiti...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_partitions_merge[True] - _pickle.PicklingError: Can't pickle local object <function test_map_partiti...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_partitions_merge[False] - _pickle.PicklingError: Can't pickle local object <function test_map_partiti...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_overlap[1] - _pickle.PicklingError: Can't pickle local object <function test_map_overlap...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_overlap[4] - _pickle.PicklingError: Can't pickle local object <function test_map_overlap...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_overlap_provide_meta - _pickle.PicklingError: Can't pickle local object <function test_map_overlap...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_map_partitions_propagates_index_metadata - _pickle.PicklingError: Can't pickle local object <function test_map_partiti...
FAILED dask/dataframe/dask_expr/tests/test_map_partitions_overlap.py::test_token_given - _pickle.PicklingError: Can't pickle local object <function test_token_given...
FAILED dask/dataframe/dask_expr/tests/test_merge.py::test_mismatching_meta - _pickle.PicklingError: Can't pickle local object <function test_mismatching...
FAILED dask/dataframe/tseries/tests/test_resample_expr.py::test_resample_agg - _pickle.PicklingError: Can't pickle local object <function test_resample_ag...
I've made a workaround to have it flip to cloudpickle more, but I'm not sure if that is correct (I don't understand why the test is only falling back with AttributeError):
diff --git a/dask/dataframe/dask_expr/tests/_util.py b/dask/dataframe/dask_expr/tests/_util.py
index 43093a709..4a49d882d 100644
--- a/dask/dataframe/dask_expr/tests/_util.py
+++ b/dask/dataframe/dask_expr/tests/_util.py
@@ -32,7 +32,7 @@ def assert_eq(a, b, *args, serialize_graph=True, **kwargs):
if hasattr(obj, "dask"):
try:
pickle.dumps(obj.dask)
- except AttributeError:
+ except (AttributeError, pickle.PicklingError):
try:
import cloudpickle as cp
Minimal Complete Verifiable Example:
python3.14 -m pytest
Anything else we need to know?:
Environment:
- Dask version: 2025.7.0
- Python version: 3.14
- Operating System: Fedora Rawhide
- Install method (conda, pip, source): source