👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Is this documented somewhere in the docs or cli help?

I've added details to the JSON tip that gets appended to all arguments that accept JSON, it looks like this:

  --kv-transfer-config KV_TRANSFER_CONFIG
                        The configurations for distributed KV cache transfer.
                        Should either be a valid JSON string or JSON keys passed individually. For example, the following sets of arguments are equivalent:
                        - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'`
                        - `--json-arg.key1 value1 --json-arg.key2.key3 value2`
                        (default: None)

Awesome and super simple code change. <3 it.

Hey @hmellor do you know if there's any chance this making it into v0.9.0?

I've asked if it can be cherry-picked!

I've asked if it can be cherry-picked!

Awesome. Thank you! That one has some other spicy updates of interest to us. Hoping it can be done.

Hey @hmellor just testing this now that 0.9.0 is out.

        - --rope-scaling.rope_type
        - "yarn"
        - --rope-scaling.factor
        - "4.0"
        - --rope-scaling.original_max_position_embeddings
        - "32768"
        - --max-model-len
        - "131072"

It's getting translated to
Overriding HF config with {'rope_scaling': {'rope-type': 'yarn', 'factor': '4.0', 'original-max-position-embeddings': '32768'}

Which is causing an error because the configurator is looking for rope_type but it's not found because the arg parse converted the _ to a dash.

Edit:
This is happening with all args that contain _ as part of their name.

Good catch, I'll make a fix on Monday!

Fix in #19030