🚀 Key Changes

This update extends LLM inference and serving capabilities to AMD GPUs, in addition to the existing NVIDIA support.

We have added shell scripts to automate ROCm driver installation, vLLM/Ollama environment setup, and model serving(vLLM). This allows users on cloud environments (Azure, AWS) with AMD GPUs to deploy LLM services without complex manual configuration.

✨ Implementation Details

1. ROCm Driver Installation (installRocmDriver.sh)

• Installs the ROCm (Radeon Open Compute) and AMD GPU driver, which is essential for using AMD GPUs.

2. vLLM Environment Deployment (deployvLLMAmd.sh)

• Configures the vLLM environment using the official rocm/vllm Docker image provided by AMD.
• Automates Docker installation and HuggingFace cache directory setup

3. vLLM Model Serving (servevLLMAmd.sh)

• Launches a specified HuggingFace model as a vLLM-powered, OpenAI-compatible API server.
• Runs in a Docker container and includes features for stable operation, such as automatic shutdown of existing servers and health checks.

💻 Test Environment and Results

✅ Azure (Radeon PRO V710)

• Result: Works Perfectly
• After installing the ROCm 7.0.1 driver, model serving and inference via vLLM/Ollama were confirmed to be running smoothly.

Ollama

vLLM