Skip to content

[Data] Improve handling of pandas.NA#53859

Merged
bveeramani merged 3 commits intomasterfrom
pandas-fix
Jun 16, 2025
Merged

[Data] Improve handling of pandas.NA#53859
bveeramani merged 3 commits intomasterfrom
pandas-fix

Conversation

@bveeramani
Copy link
Copy Markdown
Member

@bveeramani bveeramani commented Jun 16, 2025 •
edited
Loading

Why are these changes needed?

This change standardizes null handling across internal block formats by converting pd.NA to None. Since different formats (e.g., Pandas and Arrow) use different null representations, this ensures consistent semantics when converting between them. In particular, we treat pd.NA as a null value, while preserving np.nan as a distinct floating-point NaN.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@bveeramani
Initial commit
aee64df 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Copilot AI review requested due to automatic review settings June 16, 2025 19:36
@bveeramani bveeramani requested a review from a team as a code owner June 16, 2025 19:36
@bveeramani
Another commit
bfee7b1 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Copilot AI reviewed Jun 16, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances how pandas.NA values are handled by converting them to Python None when rows are materialized as dictionaries.

  • Adds a new test verifying pd.NA → None conversion in iter_rows.
  • Makes TableRow.as_pydict an abstract method.
  • Implements as_pydict in the pandas block accessor to translate pd.NA to None.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
python/ray/data/tests/test_pandas_block.py Added test_iter_rows_with_na to cover pd.NA → None conversion
python/ray/data/_internal/row.py Marked as_pydict as an abstract method on TableRow
python/ray/data/_internal/pandas_block.py Implemented as_pydict to map pd.NA values to None
Comments suppressed due to low confidence (3)

python/ray/data/tests/test_pandas_block.py:465

  • The comment mentions 'NaN values' but this test is for pd.NA. Please update it to 'NA values' for clarity.
# We should return None for NaN values.

python/ray/data/_internal/pandas_block.py:113

  • Type annotations Dict and Any are used here but not imported in this module; add from typing import Any, Dict at the top to prevent a NameError.
    def as_pydict(self) -> Dict[str, Any]:

python/ray/data/_internal/row.py:34

  • [nitpick] Using an Ellipsis stub in an abstract method can mask missing implementations. Consider raising NotImplementedError or using pass to make the intent explicit.
        ...

@bveeramani bveeramani assigned raulchen and unassigned raulchen Jun 16, 2025
@bveeramani
Remove extral ine
2282b9d 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani bveeramani enabled auto-merge (squash) June 16, 2025 22:11
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Jun 16, 2025
@bveeramani bveeramani merged commit 42c5837 into master Jun 16, 2025
6 of 7 checks passed
@bveeramani bveeramani deleted the pandas-fix branch June 16, 2025 23:46
elliot-barn pushed a commit that referenced this pull request Jun 18, 2025
@bveeramani @elliot-barn
[Data] Improve handling of pandas.NA (#53859)
e695986 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

This change standardizes null handling across internal block formats by
converting pd.NA to None. Since different formats (e.g., Pandas and
Arrow) use different null representations, this ensures consistent
semantics when converting between them. In particular, we treat pd.NA as
a null value, while preserving np.nan as a distinct floating-point NaN.
## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
minerharry pushed a commit to minerharry/ray that referenced this pull request Jun 27, 2025
@bveeramani @minerharry
[Data] Improve handling of pandas.NA (ray-project#53859)
8525878 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

This change standardizes null handling across internal block formats by
converting pd.NA to None. Since different formats (e.g., Pandas and
Arrow) use different null representations, this ensures consistent
semantics when converting between them. In particular, we treat pd.NA as
a null value, while preserving np.nan as a distinct floating-point NaN.
## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
elliot-barn pushed a commit that referenced this pull request Jul 2, 2025
@bveeramani @elliot-barn
[Data] Improve handling of pandas.NA (#53859)
04ede5d 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

This change standardizes null handling across internal block formats by
converting pd.NA to None. Since different formats (e.g., Pandas and
Arrow) use different null representations, this ensures consistent
semantics when converting between them. In particular, we treat pd.NA as
a null value, while preserving np.nan as a distinct floating-point NaN.
## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@raulchen raulchen raulchen approved these changes

Assignees

@raulchen raulchen

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bveeramani @raulchen