[2/n] [Serve] poll outbound deployments into deployment state by abrarsheikh · Pull Request #58350 · ray-project/ray

abrarsheikh · 2025-11-01T01:23:07Z

fetch outbound deployments from all replicas at initialization.

Next PR -> #58355

Signed-off-by: abrar <abrar@anyscale.com>

…yments

Signed-off-by: abrar <abrar@anyscale.com>

## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: #58350 --------- Signed-off-by: abrar <abrar@anyscale.com>

…brar-controller

Signed-off-by: abrar <abrar@anyscale.com>

python/ray/serve/_private/deployment_state.py

Signed-off-by: abrar <abrar@anyscale.com>

python/ray/serve/_private/deployment_state.py

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com>

zcin

general question: since deployment handles stored on replicas is not something that is often changed, in fact in 99% of cases it will be exactly the set that it started with, is consistent polling the best way to support this?

python/ray/serve/_private/deployment_state.py

python/ray/serve/_private/constants.py

abrarsheikh · 2025-11-13T23:44:25Z

general question: since deployment handles stored on replicas is not something that is often changed, in fact in 99% of cases it will be exactly the set that it started with, is consistent polling the best way to support this?

You are right, polling only helps with the case where user uses get_deployment_handle calls within the endpoint. Or an extreme case is when user has a get_deployment_handle behind a conditional if blocks that gets executed in 1% of the case. Polling ensures that we capture it eventually.

But i do agree this is rare. I am okay to keep this simple to begin with. wdyt?

zcin · 2025-11-13T23:48:14Z

Yeah, agree with keep simple and add support incrementally.

abrarsheikh · 2025-11-13T23:54:53Z

Yeah, agree with keep simple and add support incrementally.

changing it

…brar-controller

Signed-off-by: abrar <abrar@anyscale.com>

python/ray/serve/_private/deployment_state.py

Signed-off-by: abrar <abrar@anyscale.com>

…brar-controller

python/ray/serve/_private/deployment_state.py

Signed-off-by: abrar <abrar@anyscale.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

…58345) ## Summary Adds a new method to expose all downstream deployments that a replica calls into, enabling dependency graph construction. ## Motivation Deployments call downstream deployments via handles in two ways: 1. **Stored handles**: Passed to `__init__()` and stored as attributes → `self.model.func.remote()` 2. **Dynamic handles**: Obtained at runtime via `serve.get_deployment_handle()` → `model.func.remote()` Previously, there was no way to programmatically discover these dependencies from a running replica. ## Implementation ### Core Changes - **`ReplicaActor.list_outbound_deployments()`**: Returns `List[DeploymentID]` of all downstream deployments - Recursively inspects user callable attributes to find stored handles (including nested in dicts/lists) - Tracks dynamic handles created via `get_deployment_handle()` at runtime using a callback mechanism - **Runtime tracking**: Modified `get_deployment_handle()` to register handles when called from within a replica via `ReplicaContext._handle_registration_callback` Next PR: ray-project#58350 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

…oject#58350) fetch outbound deployments from all replicas at initialization. Next PR -> ray-project#58355 --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

abrarsheikh added 2 commits October 31, 2025 18:08

expose outbound deployment ids from replica actor

782720b

Signed-off-by: abrar <abrar@anyscale.com>

[Serve] poll outbound deployments into deployment state

50353d1

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh changed the base branch from master to dag-of-deployments November 1, 2025 01:23

abrarsheikh requested review from akyang-anyscale and zcin November 1, 2025 01:24

abrarsheikh added the go add ONLY when ready to merge, run all tests label Nov 1, 2025

abrarsheikh mentioned this pull request Nov 1, 2025

[1/n] expose outbound deployment ids from replica actor #58345

Merged

abrarsheikh added 5 commits November 1, 2025 19:20

fix type

5e9901c

Signed-off-by: abrar <abrar@anyscale.com>

change how we scan for deployment handle

ea7d807

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into dag-of-deplo…

be35418

…yments

update docstrings

297261e

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'dag-of-deployments' into SERVE-1425-abrar-controller

494fd9f

Base automatically changed from dag-of-deployments to master November 6, 2025 21:33

abrarsheikh added 2 commits November 6, 2025 21:33

Merge branch 'master' of github.com:ray-project/ray into SERVE-1425-a…

bc6a053

…brar-controller

use all replicas

c81ee07

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh marked this pull request as ready for review November 6, 2025 22:03

abrarsheikh requested a review from a team as a code owner November 6, 2025 22:03

cursor bot reviewed Nov 6, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Outdated Show resolved Hide resolved

address bot

a98c82a

Signed-off-by: abrar <abrar@anyscale.com>

cursor bot reviewed Nov 6, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Outdated Show resolved Hide resolved

ray-gardener bot added the serve Ray Serve Related Issue label Nov 7, 2025

zcin reviewed Nov 13, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Outdated Show resolved Hide resolved

python/ray/serve/_private/deployment_state.py Outdated Show resolved Hide resolved

python/ray/serve/_private/constants.py Outdated Show resolved Hide resolved

abrarsheikh added 2 commits November 14, 2025 02:32

Merge branch 'master' of github.com:ray-project/ray into SERVE-1425-a…

015c95d

…brar-controller

simplify outbound deployments

47ec98c

Signed-off-by: abrar <abrar@anyscale.com>

cursor bot reviewed Nov 14, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Show resolved Hide resolved

python/ray/serve/_private/deployment_state.py Outdated Show resolved Hide resolved

abrarsheikh added 4 commits November 14, 2025 04:34

fix test

85f89a4

Signed-off-by: abrar <abrar@anyscale.com>

sorted key

bd298ba

Signed-off-by: abrar <abrar@anyscale.com>

fix test

0bd5b64

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into SERVE-1425-a…

97df16e

…brar-controller

cursor bot reviewed Nov 14, 2025

View reviewed changes

python/ray/serve/_private/deployment_state.py Show resolved Hide resolved

filter replicas from old version

3accd0d

Signed-off-by: abrar <abrar@anyscale.com>

zcin approved these changes Nov 14, 2025

View reviewed changes

abrarsheikh merged commit 9433631 into master Nov 14, 2025
6 checks passed

abrarsheikh deleted the SERVE-1425-abrar-controller branch November 14, 2025 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[2/n] [Serve] poll outbound deployments into deployment state#58350

[2/n] [Serve] poll outbound deployments into deployment state#58350
abrarsheikh merged 17 commits intomasterfrom
SERVE-1425-abrar-controller

abrarsheikh commented Nov 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

zcin left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrarsheikh commented Nov 13, 2025

Uh oh!

zcin commented Nov 13, 2025

Uh oh!

abrarsheikh commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

abrarsheikh commented Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zcin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrarsheikh commented Nov 13, 2025

Uh oh!

zcin commented Nov 13, 2025

Uh oh!

abrarsheikh commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abrarsheikh commented Nov 1, 2025 •

edited

Loading

zcin left a comment •

edited

Loading