[Data] Rename encoder preprocessor private fields by rayhhome · Pull Request #61028 · ray-project/ray

rayhhome · 2026-02-12T20:13:32Z

Description

Rename fields in preprocessors to conform to naming convention for private fields in classes

Related issues

Fixes variable naming issue reported in #59890

Additional information

backwards compatibility with property so that old variable names still provide access to the private variables

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

… plan Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…reprocessor field Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

gemini-code-assist

Code Review

This pull request refactors several preprocessors to use private fields with a leading underscore, adding properties for backward compatibility. The changes are extensive and mostly look good. I've found a few minor issues: a typo in a setter method name in LabelEncoder, some missing return type hints in setters within scaler.py for consistency, and an inconsistency in attribute access in RobustScaler. Addressing these will improve code consistency and correctness.

python/ray/data/preprocessors/encoder.py

python/ray/data/preprocessors/scaler.py

Copilot

Pull request overview

This PR refactors Ray Data preprocessors to rename instance fields to underscore-prefixed private names (e.g. columns → _columns) while keeping public access via @property to preserve the external API and align with the project’s private-field naming convention.

Changes:

Rename many preprocessor configuration fields to private (_columns, _output_columns, etc.) and update internal references accordingly.
Add @property getters/setters to preserve the previous public attribute surface (e.g. preprocessor.columns).
Update Preprocessor to store the stat computation plan in _stat_computation_plan and expose it via a property.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
python/ray/data/preprocessors/vectorizer.py	Rename vectorizer config fields to private and add compatibility properties.
python/ray/data/preprocessors/transformer.py	Rename transformer config fields to private and add compatibility properties.
python/ray/data/preprocessors/torch.py	Add properties for existing private fields to preserve public attribute access.
python/ray/data/preprocessors/tokenizer.py	Rename tokenizer config fields to private and add compatibility properties.
python/ray/data/preprocessors/scaler.py	Rename scaler config fields to private and update stat plan access.
python/ray/data/preprocessors/normalizer.py	Rename normalizer fields to private; add type hint for `norm`.
python/ray/data/preprocessors/imputer.py	Rename imputer fields to private and update usage/serialization fields.
python/ray/data/preprocessors/hasher.py	Rename hasher fields to private and add compatibility properties.
python/ray/data/preprocessors/encoder.py	Rename encoder fields to private and update internal usage/serialization fields.
python/ray/data/preprocessors/discretizer.py	Rename CustomKBinsDiscretizer fields to private and add compatibility properties.
python/ray/data/preprocessors/chain.py	Rename `preprocessors` to `_preprocessors` and add a property for compatibility.
python/ray/data/preprocessor.py	Move stat plan to `_stat_computation_plan`, update accessors, and adjust pickling behavior.
ci/lint/pydoclint-baseline.txt	Remove an entry now satisfied by added type hints.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessor.py

python/ray/data/preprocessors/imputer.py

python/ray/data/preprocessors/hasher.py

python/ray/data/preprocessors/vectorizer.py

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessors/encoder.py

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessors/scaler.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessors/chain.py

python/ray/data/preprocessors/discretizer.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessors/hasher.py

python/ray/data/preprocessors/vectorizer.py

python/ray/data/preprocessors/chain.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessor.py

python/ray/data/preprocessors/concatenator.py

python/ray/data/preprocessors/normalizer.py

…ty + remove preprocessor setter Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessors/scaler.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…rename-private-fields

python/ray/data/preprocessor.py

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

python/ray/data/preprocessor.py

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

python/ray/data/preprocessors/discretizer.py

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

python/ray/data/preprocessors/imputer.py

python/ray/data/preprocessors/discretizer.py

python/ray/data/preprocessor.py

…rename-private-fields

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

iamjustinhsu · 2026-02-24T23:05:58Z

python/ray/data/preprocessors/imputer.py

+        if self._strategy == "constant":
            self._is_fittable = False

+    def __setstate__(self, state: Dict[str, Any]) -> None:


@rayhhome, in what context would we be dealing with an old pickled object across ray versions?

I think it would be when the user is using older versioned pickled preprocessors (like the ray_pickle.dumps in FrameworkCheckpoint). I believe ray_pickle applies standard Python unpickling, so __setstate__ would be used for restoring objects?

i see, but i see that

FrameworkCheckpoint isn't labeled as a @PublicAPI

https://github.com/anyscale/rayturbo/blob/fada4124e58de1ac14a9a33ae303edc0108d04b8/python/ray/train/predictor.py#L94 looks like loading from a checkpoint is deprecated?

Please correct me if i'm wrong. Will probably need to loop in @xinyuangui2 for context

Merged in the interest on incremental progress.

Re: the additional complexity from implementing __setstate__ -- I think we should remove Predictor later this year. If the pickling is only relevant to Predictor, then I think we should remove all of the preprocessor pickling logic with it. I think the logic is non-trivial to read and maintain

…61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to #61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: Mạnh Lê Đức <naruto12308@gmail.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: bittoby <bittoby@users.noreply.github.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com> Signed-off-by: Parag Ekbote <thecoolekbote189@gmail.com>

…ay-project#61341) ## Description The `SerializablePreprocessorBase` abstract class declares functions for saving and loading preprocessor states and should be implemented by all preprocessors. This PR implements the abstract methods for all preprocessors that are not yet inheriting from this base class. ## Related issues Related to ray-project#61028 , which implemented backwards compatibility for legacy pickling layer. The `__setstate__` functions should be removed along with the deprecated `Predictor`, while `_get_serializable_fields` and `_set_serializable_fields` should be used instead for saving and loading preprocessor states in future iterations. ## Additional information Accompanied by new field serializing tests of all preprocessors involved. --------- Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

rayhhome added 16 commits February 9, 2026 12:41

Renaming columns and output_columns to indicate them as private

a330c16

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge remote-tracking branch 'origin' into rename-private-fields

4fbad12

Preliminary rename encoder and scaler preprocessor

f5056ea

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Fix dtype attribute name mismatch

67265db

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for columns and output_columns + rename stat computation…

e4aa9a2

… plan Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for imputer fields

dc3c2f8

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Add fallback for scaler fields

1557770

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for discretizer fields

510cee7

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for vectorizer fields

e9ad2af

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Remove optional type wrapper to restrict private variable type

590173a

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for tokenizer fields

d05813b

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for transformer and normalizer fields

e24a2e6

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for hasher and add norm type annotation

179ac09

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Rename and add fallback for chain and torch

3cf0640

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into rename-private-fields

3c919bb

Revert test files, fix label encoder variable name mismatch, rename p…

e9d2209

…reprocessor field Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Copilot AI review requested due to automatic review settings February 12, 2026 20:13

rayhhome requested a review from a team as a code owner February 12, 2026 20:13

Copilot started reviewing on behalf of rayhhome February 12, 2026 20:14 View session

gemini-code-assist bot reviewed Feb 12, 2026

View reviewed changes

Copilot AI reviewed Feb 12, 2026

View reviewed changes

cursor bot reviewed Feb 12, 2026

View reviewed changes

rayhhome added 2 commits February 12, 2026 13:05

Fix omitted field names and wrong impl

9699199

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Impl setstate for tokenizer and vectorizer for backwards compatibility

9b63e09

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

alexeykudinkin reviewed Feb 12, 2026

View reviewed changes

python/ray/data/preprocessors/chain.py Outdated Show resolved Hide resolved

python/ray/data/preprocessors/discretizer.py Outdated Show resolved Hide resolved

Fix discretizer repr for backwards compatibility

876d6ee

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

cursor bot reviewed Feb 12, 2026

View reviewed changes

python/ray/data/preprocessors/hasher.py Outdated Show resolved Hide resolved

python/ray/data/preprocessors/vectorizer.py Outdated Show resolved Hide resolved

python/ray/data/preprocessors/chain.py Outdated Show resolved Hide resolved

Remove unneeded setters + rename concatenator fields

70883f1

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

cursor bot reviewed Feb 12, 2026

View reviewed changes

python/ray/data/preprocessor.py Outdated Show resolved Hide resolved

python/ray/data/preprocessors/concatenator.py Outdated Show resolved Hide resolved

python/ray/data/preprocessors/normalizer.py Show resolved Hide resolved

Impl setstate for normalizer and tranformer for backwards compatibili…

d90b279

…ty + remove preprocessor setter Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

rayhhome added 2 commits February 19, 2026 15:24

Use dataclass for field default mapping

dbdfce7

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into rename-private-fields

a24befd

goutamvenkat-anyscale approved these changes Feb 20, 2026

View reviewed changes

Merge branch 'master' into rename-private-fields

d6e8e6e

cursor bot reviewed Feb 20, 2026

View reviewed changes

python/ray/data/preprocessors/scaler.py Outdated Show resolved Hide resolved

rayhhome added 2 commits February 20, 2026 13:26

Remove stray parenthesis

637c33e

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'rename-private-fields' of github.com:rayhhome/ray into …

d8e6d11

…rename-private-fields

bveeramani added the go add ONLY when ready to merge, run all tests label Feb 20, 2026

Merge branch 'master' into rename-private-fields

72e52bb

cursor bot reviewed Feb 23, 2026

View reviewed changes

python/ray/data/preprocessor.py Show resolved Hide resolved

rayhhome added 3 commits February 23, 2026 10:11

Make sure to remove both new and legacy names

0021305

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into rename-private-fields

44f51dd

Merge branch 'master' into rename-private-fields

7a344a5

cursor bot reviewed Feb 23, 2026

View reviewed changes

python/ray/data/preprocessor.py Show resolved Hide resolved

Merge branch 'master' into rename-private-fields

de95577

cursor bot reviewed Feb 23, 2026

View reviewed changes

python/ray/data/preprocessors/discretizer.py Outdated Show resolved Hide resolved

rayhhome added 2 commits February 24, 2026 11:22

Merge branch 'master' into rename-private-fields

99fe783

Merge branch 'master' into rename-private-fields

1bc19de

cursor bot reviewed Feb 24, 2026

View reviewed changes

python/ray/data/preprocessors/imputer.py Show resolved Hide resolved

python/ray/data/preprocessors/discretizer.py Outdated Show resolved Hide resolved

python/ray/data/preprocessor.py Show resolved Hide resolved

rayhhome added 3 commits February 24, 2026 13:58

Merge branch 'rename-private-fields' of github.com:rayhhome/ray into …

bc2b772

…rename-private-fields

Apply cursor suggestions

7f7cacb

Signed-off-by: Sirui Huang <ray.huang@anyscale.com>

Merge branch 'master' into rename-private-fields

599cfc5

iamjustinhsu reviewed Feb 24, 2026

View reviewed changes

bveeramani merged commit 09c7c76 into ray-project:master Feb 25, 2026
5 checks passed

rayhhome mentioned this pull request Feb 26, 2026

[Data] Make All Preprocessors Implement SerializablePreprocessorBase #61341

Merged

Conversation

rayhhome commented Feb 12, 2026

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iamjustinhsu Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

rayhhome Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

iamjustinhsu Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bveeramani Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

iamjustinhsu Feb 25, 2026 •

edited

Loading