[serve] Serve is unable to start when tracing is enabled #57803
Description
What happened + What you expected to happen
I'm unable to get Serve start (i.e. with serve start if the cluster was initialized with _tracing_startup_hook passed.
Trace:
2025-10-16 19:57:02,236 INFO worker.py:1771 -- Connecting to existing Ray cluster at address: 127.0.0.1:55652...
2025-10-16 19:57:02,241 INFO worker.py:1942 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265
Traceback (most recent call last):
File "/Users/scorchio/Workspace/fastapi-ray/.venv/bin/serve", line 10, in <module>
sys.exit(cli())
^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/click/core.py", line 1462, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/click/core.py", line 1383, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/click/core.py", line 1850, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/click/core.py", line 1246, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/click/core.py", line 814, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/scripts.py", line 216, in start
serve.start(
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/api.py", line 113, in start
_private_api.serve_start(
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/_private/api.py", line 207, in serve_start
client = _get_global_client(_health_check_controller=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/context.py", line 94, in _get_global_client
return _connect(raise_if_no_controller_running)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/context.py", line 152, in _connect
client = ServeControllerClient(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/serve/_private/client.py", line 65, in __init__
self._http_config: HTTPOptions = ray.get(controller.get_http_config.remote())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/actor.py", line 675, in remote
return self._remote(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 439, in _start_span
return method(self, args, kwargs, *_args, **_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/actor.py", line 858, in _remote
obj_ref = invocation(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/actor.py", line 838, in invocation
return dst_actor._actor_method_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/actor.py", line 2048, in _actor_method_call
list_args = signature.flatten_args(function_signature, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/_common/signature.py", line 130, in flatten_args
validate_args(signature_parameters, args, kwargs)
File "/Users/scorchio/Workspace/fastapi-ray/.venv/lib/python3.12/site-packages/ray/_common/signature.py", line 101, in validate_args
raise TypeError(str(exc)) from None
TypeError: got an unexpected keyword argument '_ray_trace_ctx'
Versions / Dependencies
ray==2.50.0
Reproduction script
- Run the following script:
import time
import ray
ray.init(
_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing"
)
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print("Stopping the Ray cluster...")- Run
serve start
Issue Severity
High: It blocks me from completing my task.