[Serve] serve build failing for LLMConfig based applications #58485
Description
What happened + What you expected to happen
Context
I was trying to generate yaml using serve build.
I followed the quickstart doc to build the serve application. Its a barebones application with one ingress deployment and one vllm deployment.
I ran the following command
serve build serve_qwen3vl:ingress_deployment
and the command falied
Expectation
Get a valid yaml config that can be added to ray service CRD.
Logs
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/dspy/bin/serve", line 7, in <module>
sys.exit(cli())
^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/click/core.py", line 1363, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/ray/serve/scripts.py", line 867, in build
config_str += yaml.dump(
^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/__init__.py", line 253, in dump
return dump_all([data], stream, Dumper=Dumper, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/__init__.py", line 241, in dump_all
dumper.represent(data)
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 27, in represent
node = self.represent_data(data)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 210, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 121, in represent_mapping
node_value = self.represent_data(item_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 202, in represent_list
return self.represent_sequence('tag:yaml.org,2002:seq', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 95, in represent_sequence
node_item = self.represent_data(item)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 210, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 121, in represent_mapping
node_value = self.represent_data(item_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 202, in represent_list
return self.represent_sequence('tag:yaml.org,2002:seq', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 95, in represent_sequence
node_item = self.represent_data(item)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 210, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 121, in represent_mapping
node_value = self.represent_data(item_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 210, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 121, in represent_mapping
node_value = self.represent_data(item_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 61, in represent_data
node = self.yaml_representers[None](self, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ec2-user/anaconda3/envs/dspy/lib/python3.12/site-packages/yaml/representer.py", line 234, in represent_undefined
raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', <AggregationFunction.MEAN: 'mean'>)
Observations
I see that the aggregation metrics were added in this diff - f07db71
It might have been broken since then.
Versions / Dependencies
Ray version : 2.51.1
Reproduction script
from ray import serve
from ray.serve.llm import LLMConfig
from ray.serve.llm.deployment import LLMServer
from ray.serve.llm.ingress import OpenAiIngress, make_fastapi_ingress
llm_config = LLMConfig(
model_loading_config=dict(
model_id="qwen3vl-235B",
model_source="Qwen/Qwen3-VL-30B-A3B-Instruct-FP8",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=1,
aggregation_function="mean",
)
),
engine_kwargs=dict(
max_model_len=4096,
tensor_parallel_size=2,
gpu_memory_utilization=0.9,
),
)
# Deploy the application
server_options = LLMServer.get_deployment_options(llm_config)
server_deployment = serve.deployment(LLMServer).options(
**server_options).bind(llm_config)
ingress_options = OpenAiIngress.get_deployment_options(
llm_configs=[llm_config])
ingress_cls = make_fastapi_ingress(OpenAiIngress)
ingress_deployment = serve.deployment(ingress_cls).options(
**ingress_options).bind([server_deployment])
Issue Severity
Medium: It is a significant difficulty but I can work around it.