Skip to content

[WIP][Feautre, Hardware] add initial suport for ascend npu#3782

Closed
22dimensions wants to merge 1 commit intosgl-project:mainfrom
22dimensions:npu
Closed

[WIP][Feautre, Hardware] add initial suport for ascend npu#3782
22dimensions wants to merge 1 commit intosgl-project:mainfrom
22dimensions:npu

Conversation

@22dimensions
Copy link
Copy Markdown
Contributor

@22dimensions 22dimensions commented Feb 22, 2025

Motivation

see issue #3781

Modifications

  1. add new device args 'npu'
  2. import torch_npu and adapt torch related code
  3. add usage document

Checklist

Examples

python -m sglang.launch_server --model-path LLM-Research/Llama-3.2-1B-Instruct  --device npu --grammar-backend outlines --attention-backend torch_native  --port 30000 --host 0.0.0.0

image

@22dimensions 22dimensions changed the title [WIP][Feautre, Hardware] add suport for ascend npu [WIP][Feautre, Hardware] add initial suport for ascend npu Feb 22, 2025
@22dimensions
Copy link
Copy Markdown
Contributor Author

#3853 is merged, so i close this pr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant