Skip to content

[Serve] fix that rank logs are spammy#57831

Merged
abrarsheikh merged 1 commit intoray-project:masterfrom
daiping8:rank_log
Oct 17, 2025
Merged

[Serve] fix that rank logs are spammy#57831
abrarsheikh merged 1 commit intoray-project:masterfrom
daiping8:rank_log

Conversation

@daiping8
Copy link
Copy Markdown
Contributor

@daiping8 daiping8 commented Oct 17, 2025

Description

This PR changes 7 rank related INFO level logs in python/ray/serve/_private/deployment_state.py to DEBUG level. These logs primarily appear in the following scenarios:

  1. When assigning a rank to a new replica
  2. When reconfiguring a replica's rank
  3. When rank reallocation is completed
  4. When discontinuous ranks are detected and minimal reallocation is performed

These logs will only be displayed when debugging is needed, and will not affect log output during normal operation.

Related issues

Closes #57807

The logs in the screenshot from the issue have all been adjusted to DEBUG level.

…rank reassignment and configuration steps in `deployment_state.py`

Change-Id: Idc61828f1686086feb88119a1242dd9c57438d13
Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
@daiping8 daiping8 requested a review from a team as a code owner October 17, 2025 02:12
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses the issue of spammy rank-related logs by changing their level from INFO to DEBUG. The changes are applied consistently across various scenarios, such as rank assignment, reconfiguration, and reallocation. This will improve the signal-to-noise ratio in the logs during normal operation, while still providing detailed information for debugging purposes. The changes are correct and align with the stated goal. Well done.

@abrarsheikh
Copy link
Copy Markdown
Contributor

i suggest doing a manual run of the app, observe the controller logs and ensure that there are no other source of logs.

@abrarsheikh abrarsheikh added the go add ONLY when ready to merge, run all tests label Oct 17, 2025
@daiping8
Copy link
Copy Markdown
Contributor Author

i suggest doing a manual run of the app, observe the controller logs and ensure that there are no other source of logs.

Could you please share the code to reproduce the issue? Thank you.

@abrarsheikh
Copy link
Copy Markdown
Contributor

i suggest doing a manual run of the app, observe the controller logs and ensure that there are no other source of logs.

Could you please share the code to reproduce the issue? Thank you.

look at test_replica_rank.py, the logs in the GH are from the controller. so after you run serve run inspect controller logs from ray dashboard

@abrarsheikh abrarsheikh self-requested a review October 17, 2025 04:54
@ray-gardener ray-gardener bot added serve Ray Serve Related Issue observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling community-contribution Contributed by the community labels Oct 17, 2025
@daiping8
Copy link
Copy Markdown
Contributor Author

daiping8 commented Oct 17, 2025

@abrarsheikh Follow your guidance! Thanks!

  1. Now:
image

There are logs related to rank.

  1. this PR + log_level: INFO
image

There are not logs related to rank.

  1. this PR + log_level: DEBUG
image

logs related to rank changed to DEBUG.

The app code is from https://docs.ray.io/en/latest/serve/getting_started.html#running-a-ray-serve-application.

@abrarsheikh abrarsheikh merged commit edfd287 into ray-project:master Oct 17, 2025
6 checks passed
justinyeh1995 pushed a commit to justinyeh1995/ray that referenced this pull request Oct 20, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes ray-project#57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
xinyuangui2 pushed a commit to xinyuangui2/ray that referenced this pull request Oct 22, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes ray-project#57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
Signed-off-by: xgui <xgui@anyscale.com>
elliot-barn pushed a commit that referenced this pull request Oct 23, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes #57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
landscapepainter pushed a commit to landscapepainter/ray that referenced this pull request Nov 17, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes ray-project#57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
Aydin-ab pushed a commit to Aydin-ab/ray-aydin that referenced this pull request Nov 19, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes ray-project#57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Future-Outlier pushed a commit to Future-Outlier/ray that referenced this pull request Dec 7, 2025
## Description
This PR changes 7 rank related INFO level logs in
`python/ray/serve/_private/deployment_state.py` to DEBUG level. These
logs primarily appear in the following scenarios:

1. When assigning a rank to a new replica
2. When reconfiguring a replica's rank
3. When rank reallocation is completed
4. When discontinuous ranks are detected and minimal reallocation is
performed

These logs will only be displayed when debugging is needed, and will not
affect log output during normal operation.

## Related issues
Closes ray-project#57807

The logs in the screenshot from the issue have all been adjusted to
DEBUG level.

Signed-off-by: daiping8 <dai.ping88@zte.com.cn>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community go add ONLY when ready to merge, run all tests observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling serve Ray Serve Related Issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Serve] rank logs are spammy

2 participants