Looks like the first two tests passed:

test_llm_serve_correctness.py:175: TimeoutError
=========================== short test summary info ============================
FAILED test_llm_serve_correctness.py::test_llm_serve_correctness[1-2] - Timeo...
FAILED test_llm_serve_correctness.py::test_llm_serve_correctness[2-2] - Timeo...
=================== 2 failed, 2 passed in 1144.43s (0:19:04) ===================

Then we see the following CUDA error while polling for the Ray Serve LLM deployment or vLLM server to come up:

Waiting for RAY server at http://localhost:8000 to be ready...
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m ERROR 2025-05-23 00:42:04,879 default_LLMRouter fjzyw7nh 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 -- Encountered failure while handling request 700823bf-0cfe-460a-bb1c-00e25d0f3fb8
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m Traceback (most recent call last):
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/anyio/streams/memory.py", line 98, in receive
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     return self.receive_nowait()
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m            ^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise WouldBlock
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m anyio.WouldBlock
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m 
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m During handling of the above exception, another exception occurred:
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m 
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m Traceback (most recent call last):
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/middleware/base.py", line 148, in call_next
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     message = await recv_stream.receive()
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/anyio/streams/memory.py", line 118, in receive
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise EndOfStream
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m anyio.EndOfStream
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m 
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m During handling of the above exception, another exception occurred:
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m 
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m Traceback (most recent call last):
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/routers/middleware.py", line 104, in _handle_application_exceptions
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     return await call_next(request)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m            ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/middleware/base.py", line 156, in call_next
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise app_exc
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/middleware/base.py", line 141, in coro
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await self.app(scope, receive_or_disconnect, send_no_error)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise exc
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await app(scope, receive, sender)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await self.middleware_stack(scope, receive, send)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await route.handle(scope, receive, send)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await self.app(scope, receive, send)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise exc
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     await app(scope, receive, sender)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     response = await f(request)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m                ^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raw_response = await run_endpoint_function(
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     return await dependant.call(**values)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/routers/router.py", line 409, in completions
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     return await self._process_llm_request(body, is_chat=False)
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/routers/router.py", line 384, in _process_llm_request
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m     raise OpenAIHTTPException(
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m ray.llm._internal.serve.configs.openai_api_models.OpenAIHTTPException
�[36m(ServeReplica:default:LLMRouter pid=14263)�[0m INFO 2025-05-23 00:42:04,885 default_LLMRouter fjzyw7nh 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 -- POST /v1/completions 500 1798.6ms
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m 2025-05-23 00:42:04,864	INFO compiled_dag_node.py:2184 -- Tearing down compiled DAG
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m 2025-05-23 00:42:04,866	INFO compiled_dag_node.py:2189 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 4c32113b625089621f568b7007000000)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m 2025-05-23 00:42:04,866	INFO compiled_dag_node.py:2189 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 26f0d6ec0bbe6fa5c5b1499c07000000)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 2025-05-23 00:42:04,873 default_LLMDeploymentqwen-0_5b rcja4aht 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 -- Failed while streaming back a response for request (700823bf-0cfe-460a-bb1c-00e25d0f3fb8): EngineDeadError('EngineCore encountered an issue. See stack trace (above) for the root cause.')
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/utils/error_handling_utils.py", line 50, in handle_failure
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for response in async_iterator:
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/llm_server.py", line 528, in _predict
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for llm_response in self.engine.generate(llm_request):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py", line 631, in generate
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for request_output in self._stats.auto_track(results_generator):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine_stats.py", line 159, in auto_track
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for x in async_iterator:
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 306, in generate
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     out = q.get_nowait() or await q.get()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m                             ^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/output_processor.py", line 51, in get
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     raise output
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 357, in output_handler
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     outputs = await engine_core.get_output_async()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 716, in get_output_async
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     raise self._format_exception(outputs) from None
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 2025-05-23 00:42:04,876 default_LLMDeploymentqwen-0_5b rcja4aht 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 -- Encountered failure while handling request 700823bf-0cfe-460a-bb1c-00e25d0f3fb8
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/utils/error_handling_utils.py", line 50, in handle_failure
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for response in async_iterator:
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/llm_server.py", line 528, in _predict
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for llm_response in self.engine.generate(llm_request):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py", line 631, in generate
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for request_output in self._stats.auto_track(results_generator):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine_stats.py", line 159, in auto_track
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     async for x in async_iterator:
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 306, in generate
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     out = q.get_nowait() or await q.get()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m                             ^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/output_processor.py", line 51, in get
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     raise output
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 357, in output_handler
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     outputs = await engine_core.get_output_async()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 716, in get_output_async
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m     raise self._format_exception(outputs) from None
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m INFO 2025-05-23 00:42:04,878 default_LLMDeploymentqwen-0_5b rcja4aht 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 -- CALL /v1/completions OK 1771.8ms
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] EngineCore encountered a fatal error.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] Traceback (most recent call last):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 389, in run_engine_core
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     engine_core.run_busy_loop()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 413, in run_busy_loop
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     self._process_engine_step()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 438, in _process_engine_step
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     outputs = self.step_fn()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]               ^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 249, in step_with_batch_queue
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     model_output = future.result()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]                    ^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/executor/ray_distributed_executor.py", line 24, in result
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     return self.ref.get()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/compiled_dag_ref.py", line 145, in get
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     raise execution_error from None
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/compiled_dag_ref.py", line 136, in get
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     ray.get(actor_execution_loop_refs, timeout=10)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     return fn(*args, **kwargs)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     return func(*args, **kwargs)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/worker.py", line 2847, in get
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/worker.py", line 935, in get_objects
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     raise value.as_instanceof_cause()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] ray.exceptions.RayTaskError(RuntimeError): �[36mray::RayWorkerWrapper.__ray_call__()�[39m (pid=14908, ip=10.0.161.217, actor_id=26f0d6ec0bbe6fa5c5b1499c07000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7426dca840d0>)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/actor.py", line 1745, in __ray_call__
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     return fn(self, *args, **kwargs)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 251, in do_exec_tasks
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     done = tasks[operation.exec_task_idx].exec_operation(
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 783, in exec_operation
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     return self._read(overlap_gpu_communication)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 674, in _read
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     input_data = self.input_reader.read()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]                  ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/common.py", line 312, in read
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     outputs = self._read_list(timeout)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]               ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/common.py", line 385, in _read_list
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     result = c.read(min(remaining_timeout, iteration_timeout))
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/torch_tensor_nccl_channel.py", line 333, in read
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     tensors = self._gpu_data_channel.read(timeout)
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/torch_tensor_nccl_channel.py", line 629, in read
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     buf = self._nccl_group.recv(
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]           ^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/nccl_group.py", line 237, in recv
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     self._cuda_stream.synchronize()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]   File "/home/ray/anaconda3/lib/python3.11/site-packages/torch/cuda/streams.py", line 99, in synchronize
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398]     super().synchronize()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] RuntimeError: CUDA error: an illegal memory access was encountered
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [core.py:398] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m INFO 05-23 00:42:04 [ray_distributed_executor.py:127] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399] AsyncLLM output_handler failed.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399] Traceback (most recent call last):
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 357, in output_handler
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399]     outputs = await engine_core.get_output_async()
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 716, in get_output_async
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399]     raise self._format_exception(outputs) from None
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m ERROR 05-23 00:42:04 [async_llm.py:399] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m INFO 05-23 00:42:04 [async_llm.py:324] Request 700823bf-0cfe-460a-bb1c-00e25d0f3fb8 failed (engine dead).
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m INFO 05-23 00:42:04 [async_llm.py:411] Aborted request 700823bf-0cfe-460a-bb1c-00e25d0f3fb8.
�[36m(RayWorkerWrapper pid=14908)�[0m INFO 05-23 00:42:00 [gpu_model_runner.py:1686] Graph capturing finished in 18 secs, took 0.24 GiB�[32m [repeated 2x across cluster]�[0m
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m �[36m(RayWorkerWrapper pid=14908)�[0m INFO 05-23 00:41:37 [backends.py:148] Compiling a graph for general shape takes 12.95 s
�[36m(ServeReplica:default:LLMDeploymentqwen-0_5b pid=14262)�[0m �[36m(RayWorkerWrapper pid=14908)�[0m INFO 05-23 00:41:40 [monitor.py:33] torch.compile takes 15.89 s in total
�[36m(RayWorkerWrapper pid=14908)�[0m [2025-05-23 00:42:04] ERROR compiled_dag_node.py:264: Compiled DAG task exited with exception
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 251, in do_exec_tasks
�[36m(RayWorkerWrapper pid=14908)�[0m     done = tasks[operation.exec_task_idx].exec_operation(
�[36m(RayWorkerWrapper pid=14908)�[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 783, in exec_operation
�[36m(RayWorkerWrapper pid=14908)�[0m     return self._read(overlap_gpu_communication)
�[36m(RayWorkerWrapper pid=14908)�[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/dag/compiled_dag_node.py", line 674, in _read
�[36m(RayWorkerWrapper pid=14908)�[0m     input_data = self.input_reader.read()
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/common.py", line 312, in read
�[36m(RayWorkerWrapper pid=14908)�[0m     outputs = self._read_list(timeout)
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/common.py", line 385, in _read_list
�[36m(RayWorkerWrapper pid=14908)�[0m     result = c.read(min(remaining_timeout, iteration_timeout))
�[36m(RayWorkerWrapper pid=14908)�[0m              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/torch_tensor_nccl_channel.py", line 333, in read
�[36m(RayWorkerWrapper pid=14908)�[0m     tensors = self._gpu_data_channel.read(timeout)
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/torch_tensor_nccl_channel.py", line 629, in read
�[36m(RayWorkerWrapper pid=14908)�[0m     buf = self._nccl_group.recv(
�[36m(RayWorkerWrapper pid=14908)�[0m           ^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/experimental/channel/nccl_group.py", line 237, in recv
�[36m(RayWorkerWrapper pid=14908)�[0m     self._cuda_stream.synchronize()
�[36m(RayWorkerWrapper pid=14908)�[0m   File "/home/ray/anaconda3/lib/python3.11/site-packages/torch/cuda/streams.py", line 99, in synchronize
�[36m(RayWorkerWrapper pid=14908)�[0m     super().synchronize()
�[36m(RayWorkerWrapper pid=14908)�[0m RuntimeError: CUDA error: an illegal memory access was encountered
�[36m(RayWorkerWrapper pid=14908)�[0m CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[36m(RayWorkerWrapper pid=14908)�[0m For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[36m(RayWorkerWrapper pid=14908)�[0m Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[36m(RayWorkerWrapper pid=14908)�[0m 
�[36m(RayWorkerWrapper pid=14908)�[0m Destructing NCCL group on actor: Actor(RayWorkerWrapper, 26f0d6ec0bbe6fa5c5b1499c07000000)

Release test llm_serve_correctness failed #53267

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions