Here is the development roadmap for 2024 Q3. Contributions and feedback are welcome.

Server API

• Add APIs for using the inference engine in a single script without launching a separate server. See also examples.
  • [RFC] Add an LLM engine #1127
• Support most OpenAI APIs: Batch, completion, chat, embedding
  • Add support for OpenAI API : offline batch(file) processing #699
  • Add support for OpenAI API parallel sampling #640
  • Add support for logprobs in OpenAI chat API #852
  • Support more OpenAI API test #916
  • Add openai embedding API #997
• Support directly taking embedding as inputs. [Feature] Generation Inputs: input_embeds #745
• Support updating the model weights without relaunching the server. @shanyu-sys
  • [Feat] Support update weights without restart server #1157
• Support Mistral endpoint in the language frontend

Performance

• Improve time-to-first-token in streaming mode with better scheduling.
  • Optimize schedule #1339
  • Fix some online scheduling delay #1345
• Implement chunked prefill. @hnyls2002 @vikranth22446
  • Chunked prefill #800
  • Support chunked prefill when radix cache is disabled #811
  • Enable chunked prefill by default #1040
  • Mixed style of chunked prefill #1013
• Implement speculative decoding. See also a prototype.
  • [Feature] plan to support medusa? #859

Parallelism

• Support sequence parallelism for long context models.
  • Sequence Parallel #1041

Quantization

• Support W8A16, W4A16 weight-only integer quantization. @zhyncs
  • Add torchao quant (int4/int8/fp8) to llama models #1341
• Support W4A8 quantization with fp8 activation and int4 weight.
• Support fp8/fp4 KV cache quantization. int4/int8 is low priority. Currently we've supported fp8 e5m2, and we should also support fp8 e4m3. @ispobock
  • [Feature] Support fp8 e5m2 kv cache with flashinfer #1204

Observability

• Integrate Grafana / Prometheus
  • [WIP] Prometheus Metrics #1461

Model Coverage

• Support interleaved window attention (gemma). @Ying1123
  • [Feat] Add window attention for gemma-2 #1056
  • [Fix] Compatibility of window attention and cuda graph #1090
  • [Fix] Window attention compatible with RadixAttention and chunked prefill #1112
  • Save memory from interleaved attention #1151 (delayed to Q4, which is dependent on new memory manager)
• Language Models
  • Mamba models
  • Deepseek models
    • Support Deepseek MoE Model #689
    • Deepseek v2 support #693
    • Support MLA for DeepSeek-V2 with Triton - step 1 #905
    • Optimize MLA/GQA/MQA Triton decoding #1138
    • feat: fix fp8 for MLA and support bmm fp8 for DeepSeek V2 #1285
• Vision Language Models
  • LLaVA-OneVision
    • [Feat] add llava-onevision, with support for (1) siglip encoder, (2) qwen2 decoder (3) openai api compatible server. #1123
  • CogVLM2
  • Cambrian-1
  • Phi-vision
• Embedding models
  • Add llama embedding modules [unreachable code] - step 1/3 #983
  • Add io struct for embedding models [unreachable code] - step 2/3 #987
  • Add e5-mistral embedding model - step 3/3 #988
  • Add openai embedding API #997
  • Support embedding input as a list #1014
  • Support Alibaba-NLP/gte-Qwen2-7B-instruct embedding Model #1186

Hardware Coverage

• AMD support
  • [Feature, Hardware] Enable SGLang on AMD GPUs via PyTorch for ROCm #1420

Language API

• Function calling. Add tools argument in sgl.gen. See also guidance and Claudette. For OpenAI models, we can translate to their function calling API (https://platform.openai.com/docs/guides/function-calling). For local models, we can use SGLang primitives (regex, select) and constrained decoding to implement a similar workflow. Or we can interrupt the decoding process to replace it with function callings. @Yiyun-Liang
  • Function calling for OpenAI backend #573
• Support sending a full serialized SGL program to the server.
• Constraint decoding
  • [FEAT] JSON constrained support #1125

LoRA Support

• Port multi-LoRA serving from S-LoRA (Full optimizations will be in Q4 planning). @Ying1123
  • [Feature] Initial support for multi-LoRA serving #1307
  • [Feature] Support LoRA path renaming and add LoRA serving benchmarks #1433

Usage examples

• Add more usage examples (e.g., parallel JSON decoding, auto parallel decoding, Self-Discover: Large Language Models Self-Compose Reasoning Structures).

Others

• Setup CI. @zhyncs @hnyls2002 @merrymercy @Ying1123
• Documentation website.
• Compiler mode optimizations for the language. (Delayed to Q4)