[data] Support CountDistinct aggregate by my-vegetable-has-exploded · Pull Request #59030 · ray-project/ray

my-vegetable-has-exploded · 2025-11-27T08:13:04Z

Description

CountDistinct allow users to compute the number of distinct values in a column, similar to SQL's COUNT(DISTINCT ...).

Related issues

close #58252

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

gemini-code-assist

Code Review

This pull request introduces a CountDistinct aggregation, which is a useful addition. The implementation correctly leverages the existing Unique aggregation. However, I've identified a high-severity issue where the ignore_nulls parameter is not being respected due to the parent class's implementation. I've suggested a fix by overriding aggregate_block in the new CountDistinct class. Additionally, I've pointed out a minor error in the docstring and proposed corrections to the tests to ensure the ignore_nulls functionality is properly validated.

python/ray/data/aggregate.py

gemini-code-assist · 2025-11-27T08:15:52Z

python/ray/data/tests/test_groupby_e2e.py

        .groupby("A")
        .aggregate(
            Count("B", alias_name="count_b", ignore_nulls=ignore_nulls),
+            CountDistinct("B", alias_name="count_distinct_b"),


The CountDistinct aggregation in this test should be parameterized with the ignore_nulls variable to properly test both cases (True and False), just like the other aggregations in this test.

Suggested change

CountDistinct("B", alias_name="count_distinct_b"),

CountDistinct("B", alias_name="count_distinct_b", ignore_nulls=ignore_nulls),

python/ray/data/tests/test_groupby_e2e.py

gemini-code-assist · 2025-11-27T08:15:52Z

python/ray/data/tests/test_groupby_e2e.py


    aggs = [
        Count("B", alias_name="count_b", ignore_nulls=ignore_nulls),
+        CountDistinct("B", alias_name="count_distinct_b"),


To ensure the ignore_nulls functionality of CountDistinct is tested, you should pass the ignore_nulls parameter from the test's parameterization.

Suggested change

CountDistinct("B", alias_name="count_distinct_b"),

CountDistinct("B", alias_name="count_distinct_b", ignore_nulls=ignore_nulls),

python/ray/data/tests/test_groupby_e2e.py

cursor · 2025-11-27T08:17:21Z

python/ray/data/tests/test_groupby_e2e.py

        {
            "B": [
                ("count_b", lambda s: s.count() if ignore_nulls else len(s)),
+                ("count_distinct_b", lambda s: s.nunique(dropna=False)),


Bug: Test uses wrong pandas comparison for CountDistinct null handling

The test compares CountDistinct (using default ignore_nulls=True) against s.nunique(dropna=False), which have opposite semantics. With ignore_nulls=True, nulls are excluded from the count, but dropna=False tells pandas to include NaN as a unique value. Since the test data contains NaN values, this mismatch will cause test failures. The pandas comparison needs to use dropna=True to match, or CountDistinct needs to pass ignore_nulls=ignore_nulls to be consistent with other aggregations in the test.

Additional Locations (1)

python/ray/data/tests/test_groupby_e2e.py#L643-L644

cursor · 2025-11-27T08:17:21Z

python/ray/data/tests/test_groupby_e2e.py

        {
            "B": [
                ("count", lambda s: s.count() if ignore_nulls else len(s)),
+                ("count_distinct", lambda s: s.nunique(skipna=False)),


Bug: Test uses wrong parameter name for pandas nunique

The test uses s.nunique(skipna=False) but pandas Series.nunique() uses the parameter dropna, not skipna. This will raise a TypeError when the test executes. The parameter name needs to be dropna.

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

alexeykudinkin

LGTM, please address feedback from Gemini

python/ray/data/aggregate.py

alexeykudinkin · 2025-12-15T19:07:38Z

python/ray/data/tests/test_groupby_e2e.py


    aggs = [
        Count("B", alias_name="count_b", ignore_nulls=ignore_nulls),
+        CountDistinct("B", alias_name="count_distinct_b"),


Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

my-vegetable-has-exploded · 2025-12-17T06:30:08Z

Hi @alexeykudinkin, thanks for review. I have resolved problems, but ci fails. I don't find actual fail info in microcheck. Could you please re-triage this test？

goutamvenkat-anyscale · 2025-12-18T02:26:38Z

Hi @alexeykudinkin, thanks for review. I have resolved problems, but ci fails. I don't find actual fail info in microcheck. Could you please re-triage this test？

Some of the errors were transient (5xx errors). I just merge master into your branch. Those should resolve

my-vegetable-has-exploded · 2025-12-18T06:11:15Z

Hi @alexeykudinkin, thanks for review. I have resolved problems, but ci fails. I don't find actual fail info in microcheck. Could you please re-triage this test？

Some of the errors were transient (5xx errors). I just merge master into your branch. Those should resolve

Thanks @goutamvenkat-anyscale !!!

github-actions · 2026-01-01T12:25:49Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

goutamvenkat-anyscale · 2026-01-02T21:52:58Z

@my-vegetable-has-exploded

[2026-01-01T21:27:19Z] Public APIs that are NOT documented:

[2026-01-01T21:27:19Z] ray.data.aggregate.CountDistinct

Looks like you're missing some documentation on CountDistinct which is the root cause of the CI failure

Adds CountDistinct to the list of aggregation functions in the API reference to improve documentation completeness. Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

my-vegetable-has-exploded · 2026-01-04T03:09:57Z

@my-vegetable-has-exploded

[2026-01-01T21:27:19Z] Public APIs that are NOT documented:

[2026-01-01T21:27:19Z] ray.data.aggregate.CountDistinct

Looks like you're missing some documentation on CountDistinct which is the root cause of the CI failure

Done

my-vegetable-has-exploded · 2026-01-08T02:25:36Z

Thank you all!

## Description `CountDistinct` allow users to compute the number of distinct values in a column, similar to SQL's `COUNT(DISTINCT ...)`. ## Related issues close ray-project#58252 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com> Co-authored-by: Goutam <goutam@anyscale.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

## Description `CountDistinct` allow users to compute the number of distinct values in a column, similar to SQL's `COUNT(DISTINCT ...)`. ## Related issues close ray-project#58252 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com> Co-authored-by: Goutam <goutam@anyscale.com>

## Description `CountDistinct` allow users to compute the number of distinct values in a column, similar to SQL's `COUNT(DISTINCT ...)`. ## Related issues close ray-project#58252 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com> Co-authored-by: Goutam <goutam@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

my-vegetable-has-exploded added 2 commits November 27, 2025 07:54

feat(data): support CountDistinct aggregate.

d59eaff

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

fix typos.

3a1a7b9

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

my-vegetable-has-exploded requested a review from a team as a code owner November 27, 2025 08:13

gemini-code-assist bot reviewed Nov 27, 2025

View reviewed changes

cursor bot reviewed Nov 27, 2025

View reviewed changes

ray-gardener bot added data Ray Data-related issues community-contribution Contributed by the community labels Nov 27, 2025

fix nunique arg.

db80923

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

omatthew98 self-assigned this Dec 3, 2025

gvspraveen requested a review from cem-anyscale December 15, 2025 18:31

alexeykudinkin reviewed Dec 15, 2025

View reviewed changes

my-vegetable-has-exploded added 2 commits December 17, 2025 03:14

Merge remote-tracking branch 'origin/master' into count-distinct

fd34e45

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

fix ignore_nulls.

4caa708

Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

Merge branch 'master' into count-distinct

1eb0b24

github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jan 1, 2026

richardliaw added go add ONLY when ready to merge, run all tests and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Jan 1, 2026

docs(api): add CountDistinct to aggregate.rst API reference

1cd5b23

Adds CountDistinct to the list of aggregation functions in the API reference to improve documentation completeness. Signed-off-by: my-vegetable-has-exploded <wy1109468038@gmail.com>

alexeykudinkin approved these changes Jan 7, 2026

View reviewed changes

alexeykudinkin merged commit 6ee6e69 into ray-project:master Jan 7, 2026
6 checks passed

	CountDistinct("B", alias_name="count_distinct_b"),
	CountDistinct("B", alias_name="count_distinct_b", ignore_nulls=ignore_nulls),

Conversation

my-vegetable-has-exploded commented Nov 27, 2025

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot Nov 27, 2025

Choose a reason for hiding this comment

Bug: Test uses wrong pandas comparison for CountDistinct null handling

Uh oh!

cursor bot Nov 27, 2025

Choose a reason for hiding this comment

Bug: Test uses wrong parameter name for pandas nunique

Uh oh!

alexeykudinkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alexeykudinkin Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

my-vegetable-has-exploded commented Dec 17, 2025

Uh oh!

goutamvenkat-anyscale commented Dec 18, 2025

Uh oh!

my-vegetable-has-exploded commented Dec 18, 2025

Uh oh!

github-actions bot commented Jan 1, 2026

Uh oh!

goutamvenkat-anyscale commented Jan 2, 2026

[2026-01-01T21:27:19Z] Public APIs that are NOT documented:

Uh oh!

my-vegetable-has-exploded commented Jan 4, 2026

[2026-01-01T21:27:19Z] Public APIs that are NOT documented:

Uh oh!

Uh oh!

my-vegetable-has-exploded commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants