README
¶
KServe
KServe is a standardized distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes.
KServe is being used by many organizations and is a Cloud Native Computing Foundation (CNCF) incubating project.
For more details, visit the KServe website.
Why KServe?
Single platform that unifies Generative and Predictive AI inference on Kubernetes. Simple enough for quick deployments, yet powerful enough to handle enterprise-scale AI workloads with advanced features.
Features
Generative AI
- 🧠 LLM-Optimized: OpenAI-compatible inference protocol for seamless integration with large language models
- 🚅 GPU Acceleration: High-performance serving with GPU support and optimized memory management for large models
- 💾 Model Caching: Intelligent model caching to reduce loading times and improve response latency for frequently used models
- 🗂️ KV Cache Offloading: Advanced memory management with KV cache offloading to CPU/disk for handling longer sequences efficiently
- 📈 Autoscaling: Request-based autoscaling capabilities optimized for generative workload patterns
- 🔧 Hugging Face Ready: Native support for Hugging Face models with streamlined deployment workflows
Predictive AI
- 🧮 Multi-Framework: Support for TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and more
- 🔀 Intelligent Routing: Seamless request routing between predictor, transformer, and explainer components with automatic traffic management
- 🔄 Advanced Deployments: Canary rollouts, inference pipelines, and ensembles with InferenceGraph
- ⚡ Autoscaling: Request-based autoscaling with scale-to-zero for predictive workloads
- 🔍 Model Explainability: Built-in support for model explanations and feature attribution to understand prediction reasoning
- 📊 Advanced Monitoring: Enables payload logging, outlier detection, adversarial detection, and drift detection
- 💰 Cost Efficient: Scale-to-zero on expensive resources when not in use, reducing infrastructure costs
Learn More
To learn more about KServe, how to use various supported features, and how to participate in the KServe community, please follow the KServe website documentation. Additionally, we have compiled a list of presentations and demos to dive through various details.
🛠 Installation
Standalone Installation
- Standard Kubernetes Installation: Compared to Serverless Installation, this is a more lightweight installation. However, this option does not support canary deployment and request based autoscaling with scale-to-zero.
- Knative Installation: KServe by default installs Knative for serverless deployment for InferenceService.
- ModelMesh Installation: You can optionally install ModelMesh to enable high-scale, high-density and frequently-changing model serving use cases.
- Quick Installation: Install KServe on your local machine.
Kubeflow Installation
KServe is an important addon component of Kubeflow, please learn more from the Kubeflow KServe documentation. Check out the following guides for running on AWS or on OpenShift Container Platform.
🛫 Create your first InferenceService
💡 Roadmap
📘 InferenceService API Reference
🧰 Developer Guide
✍ Contributor Guide
🤝 Adopters
Star History
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
agent
command
|
|
|
crd-gen
command
|
|
|
llmisvc
command
|
|
|
localmodel
command
|
|
|
localmodelnode
command
|
|
|
manager
command
|
|
|
router
command
|
|
|
spec-gen
command
|
|
|
pkg
|
|
|
apis
Package apis contains Kubernetes API groups.
|
Package apis contains Kubernetes API groups. |
|
apis/serving/v1alpha1
Package v1alpha1 contains API Schema definitions for the serving v1alpha1 API group
|
Package v1alpha1 contains API Schema definitions for the serving v1alpha1 API group |
|
apis/serving/v1beta1
Package v1beta1 contains API Schema definitions for the serving v1beta1 API group +k8s:openapi-gen=true +kubebuilder:object:generate=true +k8s:defaulter-gen=TypeMeta +groupName=serving.kserve.io
|
Package v1beta1 contains API Schema definitions for the serving v1beta1 API group +k8s:openapi-gen=true +kubebuilder:object:generate=true +k8s:defaulter-gen=TypeMeta +groupName=serving.kserve.io |
|
client/clientset/versioned/fake
This package has the automatically generated fake clientset.
|
This package has the automatically generated fake clientset. |
|
client/clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
|
This package contains the scheme of the automatically generated clientset. |
|
client/clientset/versioned/typed/serving/v1alpha1
This package has the automatically generated typed clients.
|
This package has the automatically generated typed clients. |
|
client/clientset/versioned/typed/serving/v1alpha1/fake
Package fake has the automatically generated clients.
|
Package fake has the automatically generated clients. |
|
client/clientset/versioned/typed/serving/v1beta1
This package has the automatically generated typed clients.
|
This package has the automatically generated typed clients. |
|
client/clientset/versioned/typed/serving/v1beta1/fake
Package fake has the automatically generated clients.
|
Package fake has the automatically generated clients. |
|
controller/v1alpha1/inferencegraph
+kubebuilder:rbac:groups=serving.kserve.io,resources=inferencegraphs;inferencegraphs/finalizers,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=inferencegraphs/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.knative.dev,resources=services,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/finalizers,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/status,verbs=get;update;patch
|
+kubebuilder:rbac:groups=serving.kserve.io,resources=inferencegraphs;inferencegraphs/finalizers,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=inferencegraphs/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.knative.dev,resources=services,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/finalizers,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/status,verbs=get;update;patch |
|
controller/v1alpha1/localmodel
+kubebuilder:rbac:groups=serving.kserve.io,resources=inferenceservices,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodegroups,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelcaches,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelcaches/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes/status,verbs=get;watch +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=nodes/status,verbs=get;watch +kubebuilder:rbac:groups=core,resources=persistentvolumes,verbs=get;list;watch;create;update;patch +kubebuilder:rbac:groups=core,resources=persistentvolumeclaims,verbs=get;list;watch;create;update;patch
|
+kubebuilder:rbac:groups=serving.kserve.io,resources=inferenceservices,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodegroups,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelcaches,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelcaches/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes/status,verbs=get;watch +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=nodes/status,verbs=get;watch +kubebuilder:rbac:groups=core,resources=persistentvolumes,verbs=get;list;watch;create;update;patch +kubebuilder:rbac:groups=core,resources=persistentvolumeclaims,verbs=get;list;watch;create;update;patch |
|
controller/v1alpha1/localmodelnode
+kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodegroups,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=clusterstoragecontainers,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes/status,verbs=get;update;patch +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=nodes/status,verbs=get;watch +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=batch,resources=jobs/status,verbs=get
|
+kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodegroups,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=clusterstoragecontainers,verbs=get;list;watch +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=localmodelnodes/status,verbs=get;update;patch +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=nodes/status,verbs=get;watch +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=batch,resources=jobs/status,verbs=get |
|
controller/v1alpha1/trainedmodel
+kubebuilder:rbac:groups=serving.kserve.io,resources=trainedmodels,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=trainedmodels/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.knative.dev,resources=services,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/status,verbs=get;update;patch +kubebuilder:rbac:groups=core,resources=serviceaccounts,verbs=get +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get;update +kubebuilder:rbac:groups=core,resources=secrets,verbs=get +kubebuilder:rbac:groups=core,resources=namespaces,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=events,verbs=get;list;watch;create;update;patch;delete
|
+kubebuilder:rbac:groups=serving.kserve.io,resources=trainedmodels,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.kserve.io,resources=trainedmodels/status,verbs=get;update;patch +kubebuilder:rbac:groups=serving.knative.dev,resources=services,verbs=get;list;watch;create;update;patch;delete +kubebuilder:rbac:groups=serving.knative.dev,resources=services/status,verbs=get;update;patch +kubebuilder:rbac:groups=core,resources=serviceaccounts,verbs=get +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get;update +kubebuilder:rbac:groups=core,resources=secrets,verbs=get +kubebuilder:rbac:groups=core,resources=namespaces,verbs=get;list;watch +kubebuilder:rbac:groups=core,resources=events,verbs=get;list;watch;create;update;patch;delete |
|
qpext
module
|
|
|
tools
|
|
|
tf2openapi/cmd
command
|
|
Click to show internal directories.
Click to hide internal directories.