[Data] Progress Reporting Refactor 1 by kyuds · Pull Request #59350 · ray-project/ray

kyuds · 2025-12-10T14:13:09Z

Description

This is a series of PRs to refactor/consolidate progress reporting and to decouple it from the executor, opstates, etc.

Related issues

Split from #58173

Additional information

N/A

…ogress Signed-off-by: kyuds <kyuseung1016@gmail.com>

Signed-off-by: kyuds <kyuseung1016@gmail.com>

…om dataclass Signed-off-by: kyuds <kyuseung1016@gmail.com>

Signed-off-by: kyuds <kyuseung1016@gmail.com>

gemini-code-assist

Code Review

This pull request refactors the progress reporting mechanism in Ray Data, successfully decoupling it from the executor and operator states. The changes introduce dedicated progress manager classes (RichExecutionProgressManager and TqdmExecutionProgressManager) to centralize the logic for displaying progress, which improves modularity and allows for more flexible progress bar implementations. The removal of direct progress bar management from PhysicalOperator and OpState simplifies these core components. The changes are well-structured and consistent across the modified files, achieving the stated objective of the refactor.

python/ray/data/_internal/progress/tqdm_progress.py

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

cursor · 2025-12-10T15:02:42Z

python/ray/data/_internal/progress/tqdm_progress.py

+
+        ctx = DataContext.get_current()
+        self._enabled = ctx.enable_progress_bars
+        self._show_op_progress = self._enabled and ctx.enable_operator_progress_bars


Bug: verbose_progress option silently ignored after refactor

The refactoring removes usage of self._options.verbose_progress from the progress bar initialization. Previously, the old code in initialize_progress_bars used verbose_progress to control whether non-AllToAll (1:1) operators would show progress bars. The new progress managers (TqdmExecutionProgressManager and RichExecutionProgressManager) only check ctx.enable_operator_progress_bars, completely ignoring the verbose_progress execution option. This breaks backward compatibility for users who set verbose_progress=False expecting to hide individual operator progress bars while still seeing AllToAll operator progress. The option is still documented as functional in ExecutionOptions and used in tests like test_backpressure_e2e.py and test_sort.py.

Additional Locations (1)

python/ray/data/_internal/progress/rich_progress.py#L182-L194

Seems valid (?)

bveeramani

LGTM except for the duplicate log issue.

We're doing a branch cut tomorrow -- I can land this after the branch cut (this is a non-trivial refactor, and I want to make sure we have time to catch issues if there are any)

bveeramani · 2025-12-12T01:22:26Z

python/ray/data/_internal/progress/tqdm_progress.py

+
+        ctx = DataContext.get_current()
+        self._enabled = ctx.enable_progress_bars
+        self._show_op_progress = self._enabled and ctx.enable_operator_progress_bars


Seems valid (?)

bveeramani · 2025-12-12T01:25:30Z

python/ray/data/_internal/execution/interfaces/physical_operator.py

@@ -28,7 +28,6 @@
 from ray.data._internal.execution.interfaces.op_runtime_metrics import OpRuntimeMetrics


I'm seeing a duplicate "execution finished" message with the code on this PR.

python -c "import ray; ray.data.range(1).materialize()

PR

✔️ Dataset dataset_1_0 execution finished in 0.31 seconds: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.00/1.00 [00:00<00:00, 3.40 row/s] - ReadRange->SplitBlocks(20): Tasks: 0; Actors: 0; Queued blocks: 0 (0.0B); Resources: 0.0 CPU, 8.0B object store: 100%|████████████████████████████| 1.00/1.00 [00:00<00:00, 3.40 row/s] 2025-12-11 17:22:50,626 INFO streaming_executor.py:297 -- ✔️ Dataset dataset_1_0 execution finished in 0.31 seconds

Master

✔️ Dataset dataset_1_0 execution finished in 0.31 seconds: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.00/1.00 [00:00<00:00, 3.29 row/s] - ReadRange->SplitBlocks(20): Tasks: 0; Actors: 0; Queued blocks: 0 (0.0B); Resources: 0.0 CPU, 8.0B object store: 100%|████████████████████████████| 1.00/1.00 [00:00<00:00, 3.29 row/s]

kyuds · 2025-12-12T01:37:53Z

python/ray/data/_internal/execution/streaming_executor.py

-                    desc, exception is None
-                )
-            elif not self._use_rich_progress() and self._global_info:
-                logger.info(desc)


if you see here, we logged in master too. Don't quite understand why it didn't show up tbh (maybe it showed up before tqdm?)

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

## Description Follow up to #59350 - motivation: better abstraction for progress bars and type checking in general. ## Related issues N/A ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

## Description This is a series of PRs to refactor/consolidate progress reporting and to decouple it from the executor, opstates, etc. ## Related issues Split from ray-project#58173 ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

## Description Follow up to ray-project#59350 - motivation: better abstraction for progress bars and type checking in general. ## Related issues N/A ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

## Description This is a series of PRs to refactor/consolidate progress reporting and to decouple it from the executor, opstates, etc. ## Related issues Split from ray-project#58173 ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

## Description Follow up to ray-project#59350 - motivation: better abstraction for progress bars and type checking in general. ## Related issues N/A ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

## Description This is a series of PRs to refactor/consolidate progress reporting and to decouple it from the executor, opstates, etc. ## Related issues Split from ray-project#58173 ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

## Description Follow up to ray-project#59350 - motivation: better abstraction for progress bars and type checking in general. ## Related issues N/A ## Additional information N/A --------- Signed-off-by: kyuds <kyuseung1016@gmail.com> Signed-off-by: Daniel Shin <kyuseung1016@gmail.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

kyuds added 8 commits December 10, 2025 10:59

move progress_manager to rich_progress file, create new module for pr…

2567d12

…ogress Signed-off-by: kyuds <kyuseung1016@gmail.com>

add tqdm progress manager

21d4d90

Signed-off-by: kyuds <kyuseung1016@gmail.com>

better typing

b4cfa17

Signed-off-by: kyuds <kyuseung1016@gmail.com>

consolidate operator progress to use resource manager instead of cust…

2cc65de

…om dataclass Signed-off-by: kyuds <kyuseung1016@gmail.com>

consolidate around progress manager-style code

0c3a57d

Signed-off-by: kyuds <kyuseung1016@gmail.com>

getter/setter for tqdm_ray

5a462aa

Signed-off-by: kyuds <kyuseung1016@gmail.com>

add missing logging message

8f725d3

Signed-off-by: kyuds <kyuseung1016@gmail.com>

move missing logging message

9d09622

Signed-off-by: kyuds <kyuseung1016@gmail.com>

kyuds requested review from a team as code owners December 10, 2025 14:13

Merge branch 'master' into progress-refactor-1

8d9a42c

gemini-code-assist bot reviewed Dec 10, 2025

View reviewed changes

cursor bot reviewed Dec 10, 2025

View reviewed changes

python/ray/data/_internal/progress/tqdm_progress.py Show resolved Hide resolved

fix bug found by cursor - mismatch names

b911675

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

cursor bot reviewed Dec 10, 2025

View reviewed changes

richardliaw added data Ray Data-related issues go add ONLY when ready to merge, run all tests labels Dec 10, 2025

bveeramani reviewed Dec 12, 2025

View reviewed changes

kyuds commented Dec 12, 2025

View reviewed changes

kyuds and others added 2 commits December 12, 2025 15:13

Merge branch 'master' into progress-refactor-1

7c0d755

better close order

b7263c2

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

kyuds changed the title ~~[Data] Progress Reporting Refactor 1~~ [Please Dont Merge][Data] Progress Reporting Refactor 1 Dec 12, 2025

kyuds added 3 commits December 12, 2025 18:37

progress bar init fix

57bfae9

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

further simplify

dbf8fd0

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

fix bug in tqdm_ray

b0c1a11

Signed-off-by: Daniel Shin <kyuseung1016@gmail.com>

kyuds changed the title ~~[Please Dont Merge][Data] Progress Reporting Refactor 1~~ [Data] Progress Reporting Refactor 1 Dec 12, 2025

bveeramani enabled auto-merge (squash) December 22, 2025 17:28

bveeramani approved these changes Dec 23, 2025

View reviewed changes

codope approved these changes Dec 23, 2025

View reviewed changes

bveeramani merged commit 3cb1b7c into ray-project:master Dec 23, 2025
7 checks passed

kyuds mentioned this pull request Dec 23, 2025

[Data] Progress Reporting Refactor 2 #59629

Merged

kyuds deleted the progress-refactor-1 branch December 30, 2025 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Progress Reporting Refactor 1#59350

[Data] Progress Reporting Refactor 1#59350
bveeramani merged 15 commits intoray-project:masterfrom
kyuds:progress-refactor-1

kyuds commented Dec 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

cursor bot Dec 10, 2025

Uh oh!

bveeramani Dec 12, 2025

Uh oh!

bveeramani left a comment

Uh oh!

bveeramani Dec 12, 2025

Uh oh!

bveeramani Dec 12, 2025

Uh oh!

kyuds Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -28,7 +28,6 @@
		from ray.data._internal.execution.interfaces.op_runtime_metrics import OpRuntimeMetrics

Conversation

kyuds commented Dec 10, 2025

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

cursor bot Dec 10, 2025

Choose a reason for hiding this comment

Bug: verbose_progress option silently ignored after refactor

Uh oh!

bveeramani Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

bveeramani Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

bveeramani Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

kyuds Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Bug: `verbose_progress` option silently ignored after refactor