Skip to content

[Serve] Handle.options modifies defaults for all future handles on same endpoint #12802

@architkulkarni

Description

@architkulkarni
handle1 = client.get_handle("my_endpoint").options(call_method="my_one_time_method")
handle2 = client.get_handle("my_endpoint") # call method of handle2 is still my_one_time_method!

As illustrated above, any options that are set for a handle at any time will remain as the defaults for all future handles created on the same endpoint. This behavior is somewhat unexpected. One idea for how to resolve this is to use a router cache instead of a handle cache in the client, and have a handle be a dataclass on top of the router. Multiple handles on the same endpoint will share the same router, but can have different option sets.

Metadata

Metadata

Labels

serveRay Serve Related Issue

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions