[Data] Make All Preprocessors Implement SerializablePreprocessorBase by rayhhome · Pull Request #61341 · ray-project/ray

rayhhome · 2026-02-26T03:06:16Z

Description

The SerializablePreprocessorBase abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class.

Related issues

Related to #61028 , which implemented backwards compatibility for legacy pickling layer. The __setstate__ functions should be removed along with the deprecated Predictor, while _get_serializable_fields and _set_serializable_fields should be used instead for saving and loading preprocessor states in future iterations.

Additional information

Accompanied by new field serializing tests of all preprocessors involved.

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

… plan Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…reprocessor field Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…ty + remove preprocessor setter Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…to make-prepro-serializable Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

gemini-code-assist · 2026-02-26T03:06:20Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copilot

Pull request overview

This PR implements serialization support for preprocessors that were not yet inheriting from SerializablePreprocessorBase. The changes add the required abstract methods _get_serializable_fields() and _set_serializable_fields() to enable CloudPickle-based serialization for all preprocessors.

Changes:

Migrated 9 preprocessor classes to inherit from SerializablePreprocessorBase and implement serialization methods
Added comprehensive serialization tests for all migrated preprocessors
Updated imports and added @SerializablePreprocessor decorator with version and identifier metadata

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
python/ray/data/preprocessors/vectorizer.py	Added serialization support for HashingVectorizer and CountVectorizer
python/ray/data/preprocessors/transformer.py	Added serialization support for PowerTransformer
python/ray/data/preprocessors/torch.py	Added serialization support for TorchVisionPreprocessor
python/ray/data/preprocessors/tokenizer.py	Added serialization support for Tokenizer
python/ray/data/preprocessors/normalizer.py	Added serialization support for Normalizer
python/ray/data/preprocessors/hasher.py	Added serialization support for FeatureHasher
python/ray/data/preprocessors/discretizer.py	Added serialization support for CustomKBinsDiscretizer and UniformKBinsDiscretizer
python/ray/data/preprocessors/concatenator.py	Added serialization support for Concatenator
python/ray/data/preprocessors/chain.py	Added serialization support for Chain preprocessor
python/ray/data/tests/preprocessors/test_vectorizer.py	Added serialization tests for vectorizers
python/ray/data/tests/preprocessors/test_transformer.py	Added serialization test for PowerTransformer
python/ray/data/tests/preprocessors/test_torch.py	Added serialization test for TorchVisionPreprocessor
python/ray/data/tests/preprocessors/test_tokenizer.py	Added serialization test for Tokenizer
python/ray/data/tests/preprocessors/test_normalizer.py	Added serialization test for Normalizer
python/ray/data/tests/preprocessors/test_hasher.py	Added serialization test for FeatureHasher
python/ray/data/tests/preprocessors/test_discretizer.py	Added serialization tests for discretizers
python/ray/data/tests/preprocessors/test_concatenator.py	Added serialization test for Concatenator
python/ray/data/tests/preprocessors/test_chain.py	Added serialization test for Chain

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

python/ray/data/preprocessors/chain.py

python/ray/data/preprocessors/hasher.py

python/ray/data/preprocessors/vectorizer.py

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessors/tokenizer.py

python/ray/data/preprocessors/vectorizer.py

python/ray/data/preprocessors/hasher.py

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessors/chain.py

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

python/ray/data/preprocessors/discretizer.py

python/ray/data/tests/preprocessors/test_torch.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…to make-prepro-serializable

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessors/tokenizer.py

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: Mạnh Lê Đức <naruto12308@gmail.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: bittoby <bittoby@users.noreply.github.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

rayhhome added 30 commits February 3, 2026 19:22

Cherry pick to fix signoff issue

b7786d3

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Renaming columns and output_columns to indicate them as private

a330c16

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge remote-tracking branch 'origin' into rename-private-fields

4fbad12

Preliminary rename encoder and scaler preprocessor

f5056ea

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Fix dtype attribute name mismatch

67265db

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for columns and output_columns + rename stat computation…

e4aa9a2

… plan Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for imputer fields

dc3c2f8

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for scaler fields

1557770

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for discretizer fields

510cee7

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for vectorizer fields

e9ad2af

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Remove optional type wrapper to restrict private variable type

590173a

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for tokenizer fields

d05813b

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for transformer and normalizer fields

e24a2e6

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for hasher and add norm type annotation

179ac09

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for chain and torch

3cf0640

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into rename-private-fields

3c919bb

Revert test files, fix label encoder variable name mismatch, rename p…

e9d2209

…reprocessor field Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Fix omitted field names and wrong impl

9699199

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Impl setstate for tokenizer and vectorizer for backwards compatibility

9b63e09

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Fix discretizer repr for backwards compatibility

876d6ee

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Remove unneeded setters + rename concatenator fields

70883f1

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Impl setstate for normalizer and tranformer for backwards compatibili…

d90b279

…ty + remove preprocessor setter Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Throw exception when required fields do not exist in serialized objects

46ddff7

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Improve setstate for concatenator for backwards compatibility

48bd7de

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add setstate for chain and hasher for backwards compatibility

c02f12a

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add backwards compatibility tests

6cfa36e

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Fix minor errors

68b0d98

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Impl SerializablePreprocessor for preprocessors missing serialization

d0dcc02

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Remove unrealistic tests

78edead

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add serialization tests + practical backwards compat tests

fe689c6

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'make-prepro-serializable' of github.com:rayhhome/ray in…

dce6876

…to make-prepro-serializable Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Copilot AI review requested due to automatic review settings February 26, 2026 03:06

rayhhome requested a review from a team as a code owner February 26, 2026 03:06

Merge branch 'master' into make-prepro-serializable

9107ba0

Copilot started reviewing on behalf of rayhhome February 26, 2026 03:06 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

cursor bot reviewed Feb 26, 2026

View reviewed changes

python/ray/data/preprocessors/discretizer.py Outdated Show resolved Hide resolved

python/ray/data/tests/preprocessors/test_torch.py Outdated Show resolved Hide resolved

rayhhome added 2 commits February 25, 2026 19:34

Fix decorator ordering and uneffective tests

8ff2978

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'make-prepro-serializable' of github.com:rayhhome/ray in…

62e7d46

…to make-prepro-serializable

ray-gardener bot added the community-contribution Contributed by the community label Feb 26, 2026

rayhhome added 4 commits February 26, 2026 10:54

Merge branch 'master' into make-prepro-serializable

116bddb

Fix formatting

e3de5a4

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into make-prepro-serializable

b301e83

Merge branch 'master' into make-prepro-serializable

d426a68

goutamvenkat-anyscale added data Ray Data-related issues go add ONLY when ready to merge, run all tests labels Feb 27, 2026

rayhhome self-assigned this Feb 28, 2026

rayhhome added 5 commits February 27, 2026 16:47

Merge branch 'master' into make-prepro-serializable

490850e

Merge branch 'master' into make-prepro-serializable

303f375

Merge branch 'master' into make-prepro-serializable

642cafe

Merge branch 'master' into make-prepro-serializable

2652101

Merge branch 'master' into make-prepro-serializable

092cd73

goutamvenkat-anyscale reviewed Mar 3, 2026

View reviewed changes

python/ray/data/preprocessors/tokenizer.py Show resolved Hide resolved

goutamvenkat-anyscale approved these changes Mar 4, 2026

View reviewed changes

bveeramani merged commit b9cbe1f into ray-project:master Mar 4, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Make All Preprocessors Implement SerializablePreprocessorBase#61341

[Data] Make All Preprocessors Implement SerializablePreprocessorBase#61341
bveeramani merged 55 commits intoray-project:masterfrom
rayhhome:make-prepro-serializable

rayhhome commented Feb 26, 2026

Uh oh!

gemini-code-assist bot commented Feb 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

rayhhome commented Feb 26, 2026

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot commented Feb 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants