update docs for custom request router and policy by abrarsheikh · Pull Request #58170 · ray-project/ray

abrarsheikh · 2025-10-25T21:10:41Z

Description

Update docs
catch exception and redirect users to docs

Related issues

Additional information

Hard to write tests for this situation.

Manually verified that this is the right exception to catch

try:
    cloudpickle.loads(b'\x80\x05\x95\xbc\x03\x00\x00\x00\x00\x00\x00\x8c\x1bray.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x03K\x03KCCnt\x00\xa0\x01\xa1\x00}\x01|\x01j\x02}\x02t\x03t\x04\x83\x01\x01\x00d\x01|\x02\x04\x00\x03\x00k\x01r*d\x02k\x00r:n\x04\x01\x00n\x0cd\x03d\x04d\x05i\x01f\x02S\x00d\x06|\x02\x04\x00\x03\x00k\x01rNd\x07k\x00r^n\x04\x01\x00n\x0cd\x08d\x04d\ti\x01f\x02S\x00d\nd\x04d\x0bi\x01f\x02S\x00d\x00S\x00\x94(NK\tK\x11K\x02\x8c\x06reason\x94\x8c\x0eBusiness hours\x94K\x12K\x14K\x04\x8c\x18Evening batch processing\x94K\x01\x8c\x0eOff-peak hours\x94t\x94(\x8c\x08datetime\x94\x8c\x03now\x94\x8c\x04hour\x94\x8c\x05print\x94\x8c\x04avro\x94t\x94\x8c\x03ctx\x94\x8c\x0ccurrent_time\x94\x8c\x0ccurrent_hour\x94\x87\x94\x8c\x1b/home/ubuntu/apps/policy.py\x94\x8c!scheduled_batch_processing_policy\x94K\x0eC\x10\x00\x03\x08\x01\x06\x01\x08\x02\x18\x01\x0c\x02\x18\x01\x0c\x03\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x18uNNNt\x94R\x94h\x00\x8c\x12_function_setstate\x94\x93\x94h#}\x94}\x94(h\x1fh\x19\x8c\x0c__qualname__\x94h\x19\x8c\x0f__annotations__\x94}\x94(h\x14\x8c\x10ray.serve.config\x94\x8c\x12AutoscalingContext\x94\x93\x94\x8c\x06return\x94h\x04\x8c\x0cGenericAlias\x94\x85\x94R\x94\x8c\x08builtins\x94\x8c\x05tuple\x94\x93\x94h2\x8c\x03int\x94\x93\x94\x8c\t_operator\x94\x8c\x07getitem\x94\x93\x94\x8c\x06typing\x94\x8c\x04Dict\x94\x93\x94h2\x8c\x03str\x94\x93\x94h:\x8c\x03Any\x94\x93\x94\x86\x94\x86\x94R\x94\x86\x94\x86\x94R\x94u\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h \x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94(h\x0eh\x0e\x8c\x08datetime\x94\x93\x94h\x12h\x00\x8c\tsubimport\x94\x93\x94h\x12\x85\x94R\x94uu\x86\x94\x86R0.')
except (ModuleNotFoundError, ImportError) as e:
    print(f"caused by {e} {type(e)}")

❯ python policy.py
caused by No module named 'avro' <class 'ModuleNotFoundError'>

Signed-off-by: abrar <abrar@anyscale.com>

angelinalg

just a couple of style nits.

doc/source/serve/advanced-guides/advanced-autoscaling.md

doc/source/serve/advanced-guides/custom-request-router.md

eicherseiji

Thanks @abrarsheikh!

For my understanding: because we also need to support Protobuf serialization, we can't do something like vLLM here and "serialize by value" to pass custom routers/policies, right?

doc/source/serve/advanced-guides/custom-request-router.md

…brar-docs

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh · 2025-10-27T19:34:17Z

"serialize by value"

this is exactly what we also doing https://github.com/ray-project/ray/blob/master/python/ray/serve/config.py#L237-L239

eicherseiji · 2025-10-27T19:35:40Z

this is exactly what we also doing https://github.com/ray-project/ray/blob/master/python/ray/serve/config.py#L237-L239

Oh duh. Okay makes sense, thanks for the reminder :)

…brar-docs

doc/source/serve/advanced-guides/custom-request-router.md

Signed-off-by: abrar <abrar@anyscale.com>

## Description 1. Update docs 2. catch exception and redirect users to docs ## Related issues ray-project#56855 ## Additional information Hard to write tests for this situation. Manually verified that this is the right exception to catch ``` try: cloudpickle.loads(b'\x80\x05\x95\xbc\x03\x00\x00\x00\x00\x00\x00\x8c\x1bray.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x03K\x03KCCnt\x00\xa0\x01\xa1\x00}\x01|\x01j\x02}\x02t\x03t\x04\x83\x01\x01\x00d\x01|\x02\x04\x00\x03\x00k\x01r*d\x02k\x00r:n\x04\x01\x00n\x0cd\x03d\x04d\x05i\x01f\x02S\x00d\x06|\x02\x04\x00\x03\x00k\x01rNd\x07k\x00r^n\x04\x01\x00n\x0cd\x08d\x04d\ti\x01f\x02S\x00d\nd\x04d\x0bi\x01f\x02S\x00d\x00S\x00\x94(NK\tK\x11K\x02\x8c\x06reason\x94\x8c\x0eBusiness hours\x94K\x12K\x14K\x04\x8c\x18Evening batch processing\x94K\x01\x8c\x0eOff-peak hours\x94t\x94(\x8c\x08datetime\x94\x8c\x03now\x94\x8c\x04hour\x94\x8c\x05print\x94\x8c\x04avro\x94t\x94\x8c\x03ctx\x94\x8c\x0ccurrent_time\x94\x8c\x0ccurrent_hour\x94\x87\x94\x8c\x1b/home/ubuntu/apps/policy.py\x94\x8c!scheduled_batch_processing_policy\x94K\x0eC\x10\x00\x03\x08\x01\x06\x01\x08\x02\x18\x01\x0c\x02\x18\x01\x0c\x03\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x18uNNNt\x94R\x94h\x00\x8c\x12_function_setstate\x94\x93\x94h#}\x94}\x94(h\x1fh\x19\x8c\x0c__qualname__\x94h\x19\x8c\x0f__annotations__\x94}\x94(h\x14\x8c\x10ray.serve.config\x94\x8c\x12AutoscalingContext\x94\x93\x94\x8c\x06return\x94h\x04\x8c\x0cGenericAlias\x94\x85\x94R\x94\x8c\x08builtins\x94\x8c\x05tuple\x94\x93\x94h2\x8c\x03int\x94\x93\x94\x8c\t_operator\x94\x8c\x07getitem\x94\x93\x94\x8c\x06typing\x94\x8c\x04Dict\x94\x93\x94h2\x8c\x03str\x94\x93\x94h:\x8c\x03Any\x94\x93\x94\x86\x94\x86\x94R\x94\x86\x94\x86\x94R\x94u\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h \x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94(h\x0eh\x0e\x8c\x08datetime\x94\x93\x94h\x12h\x00\x8c\tsubimport\x94\x93\x94h\x12\x85\x94R\x94uu\x86\x94\x86R0.') except (ModuleNotFoundError, ImportError) as e: print(f"caused by {e} {type(e)}") ``` ``` ❯ python policy.py caused by No module named 'avro' <class 'ModuleNotFoundError'> ``` --------- Signed-off-by: abrar <abrar@anyscale.com>

## Description 1. Update docs 2. catch exception and redirect users to docs ## Related issues ray-project#56855 ## Additional information Hard to write tests for this situation. Manually verified that this is the right exception to catch ``` try: cloudpickle.loads(b'\x80\x05\x95\xbc\x03\x00\x00\x00\x00\x00\x00\x8c\x1bray.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x03K\x03KCCnt\x00\xa0\x01\xa1\x00}\x01|\x01j\x02}\x02t\x03t\x04\x83\x01\x01\x00d\x01|\x02\x04\x00\x03\x00k\x01r*d\x02k\x00r:n\x04\x01\x00n\x0cd\x03d\x04d\x05i\x01f\x02S\x00d\x06|\x02\x04\x00\x03\x00k\x01rNd\x07k\x00r^n\x04\x01\x00n\x0cd\x08d\x04d\ti\x01f\x02S\x00d\nd\x04d\x0bi\x01f\x02S\x00d\x00S\x00\x94(NK\tK\x11K\x02\x8c\x06reason\x94\x8c\x0eBusiness hours\x94K\x12K\x14K\x04\x8c\x18Evening batch processing\x94K\x01\x8c\x0eOff-peak hours\x94t\x94(\x8c\x08datetime\x94\x8c\x03now\x94\x8c\x04hour\x94\x8c\x05print\x94\x8c\x04avro\x94t\x94\x8c\x03ctx\x94\x8c\x0ccurrent_time\x94\x8c\x0ccurrent_hour\x94\x87\x94\x8c\x1b/home/ubuntu/apps/policy.py\x94\x8c!scheduled_batch_processing_policy\x94K\x0eC\x10\x00\x03\x08\x01\x06\x01\x08\x02\x18\x01\x0c\x02\x18\x01\x0c\x03\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x18uNNNt\x94R\x94h\x00\x8c\x12_function_setstate\x94\x93\x94h#}\x94}\x94(h\x1fh\x19\x8c\x0c__qualname__\x94h\x19\x8c\x0f__annotations__\x94}\x94(h\x14\x8c\x10ray.serve.config\x94\x8c\x12AutoscalingContext\x94\x93\x94\x8c\x06return\x94h\x04\x8c\x0cGenericAlias\x94\x85\x94R\x94\x8c\x08builtins\x94\x8c\x05tuple\x94\x93\x94h2\x8c\x03int\x94\x93\x94\x8c\t_operator\x94\x8c\x07getitem\x94\x93\x94\x8c\x06typing\x94\x8c\x04Dict\x94\x93\x94h2\x8c\x03str\x94\x93\x94h:\x8c\x03Any\x94\x93\x94\x86\x94\x86\x94R\x94\x86\x94\x86\x94R\x94u\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h \x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94(h\x0eh\x0e\x8c\x08datetime\x94\x93\x94h\x12h\x00\x8c\tsubimport\x94\x93\x94h\x12\x85\x94R\x94uu\x86\x94\x86R0.') except (ModuleNotFoundError, ImportError) as e: print(f"caused by {e} {type(e)}") ``` ``` ❯ python policy.py caused by No module named 'avro' <class 'ModuleNotFoundError'> ``` --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

## Description 1. Update docs 2. catch exception and redirect users to docs ## Related issues ray-project#56855 ## Additional information Hard to write tests for this situation. Manually verified that this is the right exception to catch ``` try: cloudpickle.loads(b'\x80\x05\x95\xbc\x03\x00\x00\x00\x00\x00\x00\x8c\x1bray.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x03K\x03KCCnt\x00\xa0\x01\xa1\x00}\x01|\x01j\x02}\x02t\x03t\x04\x83\x01\x01\x00d\x01|\x02\x04\x00\x03\x00k\x01r*d\x02k\x00r:n\x04\x01\x00n\x0cd\x03d\x04d\x05i\x01f\x02S\x00d\x06|\x02\x04\x00\x03\x00k\x01rNd\x07k\x00r^n\x04\x01\x00n\x0cd\x08d\x04d\ti\x01f\x02S\x00d\nd\x04d\x0bi\x01f\x02S\x00d\x00S\x00\x94(NK\tK\x11K\x02\x8c\x06reason\x94\x8c\x0eBusiness hours\x94K\x12K\x14K\x04\x8c\x18Evening batch processing\x94K\x01\x8c\x0eOff-peak hours\x94t\x94(\x8c\x08datetime\x94\x8c\x03now\x94\x8c\x04hour\x94\x8c\x05print\x94\x8c\x04avro\x94t\x94\x8c\x03ctx\x94\x8c\x0ccurrent_time\x94\x8c\x0ccurrent_hour\x94\x87\x94\x8c\x1b/home/ubuntu/apps/policy.py\x94\x8c!scheduled_batch_processing_policy\x94K\x0eC\x10\x00\x03\x08\x01\x06\x01\x08\x02\x18\x01\x0c\x02\x18\x01\x0c\x03\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x18uNNNt\x94R\x94h\x00\x8c\x12_function_setstate\x94\x93\x94h#}\x94}\x94(h\x1fh\x19\x8c\x0c__qualname__\x94h\x19\x8c\x0f__annotations__\x94}\x94(h\x14\x8c\x10ray.serve.config\x94\x8c\x12AutoscalingContext\x94\x93\x94\x8c\x06return\x94h\x04\x8c\x0cGenericAlias\x94\x85\x94R\x94\x8c\x08builtins\x94\x8c\x05tuple\x94\x93\x94h2\x8c\x03int\x94\x93\x94\x8c\t_operator\x94\x8c\x07getitem\x94\x93\x94\x8c\x06typing\x94\x8c\x04Dict\x94\x93\x94h2\x8c\x03str\x94\x93\x94h:\x8c\x03Any\x94\x93\x94\x86\x94\x86\x94R\x94\x86\x94\x86\x94R\x94u\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h \x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94(h\x0eh\x0e\x8c\x08datetime\x94\x93\x94h\x12h\x00\x8c\tsubimport\x94\x93\x94h\x12\x85\x94R\x94uu\x86\x94\x86R0.') except (ModuleNotFoundError, ImportError) as e: print(f"caused by {e} {type(e)}") ``` ``` ❯ python policy.py caused by No module named 'avro' <class 'ModuleNotFoundError'> ``` --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

## Description 1. Update docs 2. catch exception and redirect users to docs ## Related issues ray-project#56855 ## Additional information Hard to write tests for this situation. Manually verified that this is the right exception to catch ``` try: cloudpickle.loads(b'\x80\x05\x95\xbc\x03\x00\x00\x00\x00\x00\x00\x8c\x1bray.cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x03K\x03KCCnt\x00\xa0\x01\xa1\x00}\x01|\x01j\x02}\x02t\x03t\x04\x83\x01\x01\x00d\x01|\x02\x04\x00\x03\x00k\x01r*d\x02k\x00r:n\x04\x01\x00n\x0cd\x03d\x04d\x05i\x01f\x02S\x00d\x06|\x02\x04\x00\x03\x00k\x01rNd\x07k\x00r^n\x04\x01\x00n\x0cd\x08d\x04d\ti\x01f\x02S\x00d\nd\x04d\x0bi\x01f\x02S\x00d\x00S\x00\x94(NK\tK\x11K\x02\x8c\x06reason\x94\x8c\x0eBusiness hours\x94K\x12K\x14K\x04\x8c\x18Evening batch processing\x94K\x01\x8c\x0eOff-peak hours\x94t\x94(\x8c\x08datetime\x94\x8c\x03now\x94\x8c\x04hour\x94\x8c\x05print\x94\x8c\x04avro\x94t\x94\x8c\x03ctx\x94\x8c\x0ccurrent_time\x94\x8c\x0ccurrent_hour\x94\x87\x94\x8c\x1b/home/ubuntu/apps/policy.py\x94\x8c!scheduled_batch_processing_policy\x94K\x0eC\x10\x00\x03\x08\x01\x06\x01\x08\x02\x18\x01\x0c\x02\x18\x01\x0c\x03\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x18uNNNt\x94R\x94h\x00\x8c\x12_function_setstate\x94\x93\x94h#}\x94}\x94(h\x1fh\x19\x8c\x0c__qualname__\x94h\x19\x8c\x0f__annotations__\x94}\x94(h\x14\x8c\x10ray.serve.config\x94\x8c\x12AutoscalingContext\x94\x93\x94\x8c\x06return\x94h\x04\x8c\x0cGenericAlias\x94\x85\x94R\x94\x8c\x08builtins\x94\x8c\x05tuple\x94\x93\x94h2\x8c\x03int\x94\x93\x94\x8c\t_operator\x94\x8c\x07getitem\x94\x93\x94\x8c\x06typing\x94\x8c\x04Dict\x94\x93\x94h2\x8c\x03str\x94\x93\x94h:\x8c\x03Any\x94\x93\x94\x86\x94\x86\x94R\x94\x86\x94\x86\x94R\x94u\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h \x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94(h\x0eh\x0e\x8c\x08datetime\x94\x93\x94h\x12h\x00\x8c\tsubimport\x94\x93\x94h\x12\x85\x94R\x94uu\x86\x94\x86R0.') except (ModuleNotFoundError, ImportError) as e: print(f"caused by {e} {type(e)}") ``` ``` ❯ python policy.py caused by No module named 'avro' <class 'ModuleNotFoundError'> ``` --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

update docs for custom request router and policy

8342a90

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added the go add ONLY when ready to merge, run all tests label Oct 25, 2025

abrarsheikh marked this pull request as ready for review October 27, 2025 16:11

abrarsheikh requested review from a team as code owners October 27, 2025 16:11

abrarsheikh requested review from akshay-anyscale and angelinalg October 27, 2025 16:12

angelinalg approved these changes Oct 27, 2025

View reviewed changes

doc/source/serve/advanced-guides/advanced-autoscaling.md Outdated Show resolved Hide resolved

doc/source/serve/advanced-guides/custom-request-router.md Outdated Show resolved Hide resolved

eicherseiji reviewed Oct 27, 2025

View reviewed changes

doc/source/serve/advanced-guides/custom-request-router.md Outdated Show resolved Hide resolved

doc/source/serve/advanced-guides/custom-request-router.md Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:ray-project/ray into SERVE-1134-a…

f411bf8

…brar-docs

ray-gardener bot added serve Ray Serve Related Issue docs An issue or change related to documentation labels Oct 27, 2025

abrarsheikh added 2 commits October 27, 2025 19:29

simplify action

8715245

Signed-off-by: abrar <abrar@anyscale.com>

simplify action

f932ec7

Signed-off-by: abrar <abrar@anyscale.com>

eicherseiji approved these changes Oct 27, 2025

View reviewed changes

Merge branch 'master' of github.com:ray-project/ray into SERVE-1134-a…

477ef29

…brar-docs

akshay-anyscale reviewed Oct 27, 2025

View reviewed changes

doc/source/serve/advanced-guides/custom-request-router.md Outdated Show resolved Hide resolved

akshay-anyscale approved these changes Oct 27, 2025

View reviewed changes

update location of policy

cf8fc0f

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh merged commit 945b535 into master Oct 28, 2025
6 checks passed

abrarsheikh deleted the SERVE-1134-abrar-docs branch October 28, 2025 00:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update docs for custom request router and policy#58170

update docs for custom request router and policy#58170
abrarsheikh merged 6 commits intomasterfrom
SERVE-1134-abrar-docs

abrarsheikh commented Oct 25, 2025

Uh oh!

angelinalg left a comment

Uh oh!

Uh oh!

Uh oh!

eicherseiji left a comment

Uh oh!

Uh oh!

Uh oh!

abrarsheikh commented Oct 27, 2025

Uh oh!

eicherseiji commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

abrarsheikh commented Oct 25, 2025

Description

Related issues

Additional information

Uh oh!

angelinalg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eicherseiji left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

abrarsheikh commented Oct 27, 2025

Uh oh!

eicherseiji commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants