Skip to content

Add SGLang provider#771

Merged
virajmehta merged 3 commits intomainfrom
aaron/sglang-provider
Jan 16, 2025
Merged

Add SGLang provider#771
virajmehta merged 3 commits intomainfrom
aaron/sglang-provider

Conversation

@Aaron1011
Copy link
Copy Markdown
Member

@Aaron1011 Aaron1011 commented Jan 15, 2025

This is heavily based on the existing OpenAI provider. Current limitations:

  • SGLang only support JSON output when a json schema is provided, so both 'json_mode = on' and 'json_mode = strict' require a schema to be provided
  • Tool use was just implemneted in [Feature] Function Calling sgl-project/sglang#2544, which is not yet released. For now, we pass through tool parameters using the standard OpenAI API, but log a warning.
  • Only chat completions (with/without streaming) are currently implemented.

Important

Add SGLang provider for chat completions with JSON schema support and integrate into deployment, configuration, UI, and testing.

  • Behavior:
    • Adds SGLangProvider to handle chat completions in sglang.rs.
    • Supports JSON output only with a provided schema.
    • Logs a warning for tool parameters as tool use is not yet released.
  • Deployment:
    • Adds Dockerfile, default.conf, entrypoint.sh, and README.md for SGLang NGINX deployment.
  • Configuration:
    • Updates merge-queue.yml to include SGLANG_API_KEY.
    • Adds SGLang configuration to tensorzero.toml.
  • UI:
    • Updates ModelBadge.tsx and ModelBadges.tsx to include SGLang styling.
  • Testing:
    • Adds E2E tests for SGLang in sglang.rs and updates mod.rs to include SGLang.

This description was created by Ellipsis for fb1eba7. It will automatically update as commits are pushed.

@Aaron1011 Aaron1011 force-pushed the aaron/sglang-provider branch 3 times, most recently from fcd3e37 to 56de86b Compare January 15, 2025 20:47
Copy link
Copy Markdown
Member

@virajmehta virajmehta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work I think a few small things and this is good to go!

Comment thread gateway/src/inference/providers/sglang.rs
Comment thread gateway/src/inference/providers/sglang.rs Outdated
Comment thread gateway/src/inference/providers/sglang.rs
Comment thread gateway/tests/e2e/providers/sglang.rs
@Aaron1011 Aaron1011 force-pushed the aaron/sglang-provider branch 3 times, most recently from 588f572 to 52f5600 Compare January 16, 2025 14:31
@Aaron1011 Aaron1011 marked this pull request as ready for review January 16, 2025 18:18
Comment thread gateway/fixtures/deployment/sgl-nginx/Dockerfile
Comment thread gateway/src/inference/providers/sglang.rs Outdated
Comment thread gateway/src/inference/providers/sglang.rs Outdated
This is heavily based on the existing OpenAI provider.
Current limitations:
* SGLang only support JSON output when a json schema is provided,
  so both 'json_mode = on' and 'json_mode = strict' require a schema
  to be provided
* Tool use was just implemneted in sgl-project/sglang#2544,
  which is not yet released. For now, we pass through tool parameters using
  the standard OpenAI API, but log a warning.
* Only chat completions (with/without streaming) are currently implemented.
@Aaron1011 Aaron1011 force-pushed the aaron/sglang-provider branch from 52f5600 to 907bac3 Compare January 16, 2025 18:49
Comment thread gateway/fixtures/deployment/sgl-nginx/Dockerfile
virajmehta
virajmehta previously approved these changes Jan 16, 2025
Copy link
Copy Markdown
Member

@virajmehta virajmehta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@virajmehta virajmehta enabled auto-merge January 16, 2025 19:52
@virajmehta virajmehta added this pull request to the merge queue Jan 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jan 16, 2025
@virajmehta virajmehta added this pull request to the merge queue Jan 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jan 16, 2025
@Aaron1011 Aaron1011 added this pull request to the merge queue Jan 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jan 16, 2025
@Aaron1011 Aaron1011 added this pull request to the merge queue Jan 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jan 16, 2025
@Aaron1011 Aaron1011 added this pull request to the merge queue Jan 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jan 16, 2025
@virajmehta virajmehta added this pull request to the merge queue Jan 16, 2025
Merged via the queue into main with commit c54b50f Jan 16, 2025
@virajmehta virajmehta deleted the aaron/sglang-provider branch January 16, 2025 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants