GKE AI Labs

Deploying a containerized agent built with the Google Agent Development Kit (ADK) that uses VertexAI API

Mon, 01 Jan 0001 00:00:00 +0000

This page was moved to https://cloud.google.com/kubernetes-engine/docs/tutorials/agentic-adk-vertex

You will be redirected in 5 seconds...

Building Agents with Agent Development Kit (ADK) on GKE Autopilot cluster using Self-Hosted LLM

Mon, 01 Jan 0001 00:00:00 +0000

This page was moved to https://cloud.google.com/kubernetes-engine/docs/tutorials/agentic-adk-vllm

You will be redirected in 5 seconds...

GKE at 65,000 Nodes: Simulated AI Workload Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes the benchmark of Google Kubernetes Engine (GKE) at a massive scale (65,000 nodes) with simulated AI workloads, using Terraform for infrastructure automation and ClusterLoader2 for performance testing.

The findings from this benchmark were published on the Google Cloud Blog.

Introduction

This benchmark simulates mixed AI workloads, specifically AI training and AI inference, on a 65,000-node GKE cluster. It focuses on evaluating the performance and scalability of the Kubernetes control plane under demanding conditions, characterized by a high number of nodes and dynamic workload changes.

Building Agents with Agent Development Kit (ADK) on GKE using Ray Serve for Self-Hosted LLMs

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial demonstrates how to deploy the Llama-3.1-8B-Instruct model on Google Kubernetes Engine (GKE) using Ray Serve and vLLM for efficient inference. Additionally, it shows how to integrate an ADK agent to interact with the model, supporting both basic chat completions and tool usage. The setup leverages a GKE Standard cluster with GPU-enabled nodes to handle the computational requirements.

By the end of this tutorial, you will:

Set up a GKE Autopilot cluster with a GPU node pool.
Deploy the Llama-3.1-8B-Instruct model using Ray Serve and vLLM.
Deploy an ADK agent that communicates with the Ray Serve endpoint.
Test the setup with basic chat completion and tool usage scenarios.
Monitor and debug the deployment using Ray Dashboard and Kubernetes logs.

Prerequisites

A terminal with kubectl, helm and gcloud and terraform 1.12.1 installed.
A Hugging Face account with a token that has Read permission to access the Llama-3.1-8B-Instruct model.
Sufficient GPU quota in your Google Cloud project. You need at least 2 NVIDIA L4 GPUs in the region (us-central1) to deploy the tutorial’s setup without quota-related errors. See About GPUs and Allocation quotas.
Access to the code repository: https://github.com/ai-on-gke/tutorials-and-examples

Filesystem structure

ray-serve/
├── adk_agent/                 # ADK agent deployment and cloudbuild configuration
| ├── example_agent/           # Agent code
|   ├── _init_.py
|   ├── agent-deployment.yaml
|   ├── agent.py
| ├── cloudbuild.yml
| ├── Dockerfile
| ├── main.py
| ├── requirements.txt
| ├── ray-serve-vllm/           # Ray image cloudbuild and service manifest
|   ├── cloudbuild.yml
|   ├── Dockerfile
|   ├── ray-service.yaml
|   ├── requirements.txt
|   ├── serve-chat-completion.py
| ├── terarform/
|   ├── artifact_registry.tf
|   ├── default_env.tfvars
|   ├── main.tf
|   ├── network.tf
|   ├── outputs.tf
|   ├── providers.tf
|   ├── variables.tf
|   └── workdload_identity.tf

Step 1: Set Up the Infrastructure with Terraform

Start by setting up the GKE cluster, service account, IAM roles, and Artifact Registry using Terraform.

Building a Movie Recommendation RAG with Agentic LlamaIndex on GKE

Mon, 01 Jan 0001 00:00:00 +0000

LlamaIndex bridges the gap between LLMs and domain-specific datasets, enabling efficient indexing and querying of unstructured data for intelligent, data-driven responses. When deployed on a GKE cluster, it ensures scalability and security by leveraging containerized workloads, GPU-based nodes, and seamless cloud-native integrations, making it ideal for ML-powered applications like RAG systems.

Overview

This tutorial will guide you through creating a Intelligent Agent & robust Retrieval-Augmented Generation (RAG) system using LlamaIndex and deploying it on Google Kubernetes Engine (GKE). The system indexes a dataset of 1000 top-rated movies and uses an LLM to generate recommendations based on user queries (e.g., “best drama movies” or “movies like Inception”).

Creating Inference Checkpoints

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This document outlines the process for converting inference checkpoints for use with various model servers, such as Jetstream with MaxText or Pytorch/XLA backends. The core of this process utilizes the checkpoint_entrypoint.sh script, packaged within a Docker container, to handle the specific conversion steps required by different server configurations. The goal is to prepare your trained model checkpoints for efficient deployment and inference serving.

GKE cross region capacity chasing with SkyPilot

Mon, 01 Jan 0001 00:00:00 +0000

Due to the limited availability of accelerator resources, customers face significant challenges in securing sufficient capacity to run their AI/ML workloads. They often require:

Preferences for VM families and accelerators, with the ability to automatically fail over to alternative configurations if their preferred resources are unavailable.
Automatic capacity acquisition across regions to address scenarios where a specific region lacks sufficient resources.

In this tutorial, we will demonstrate how to leverage the open-source software SkyPilot to help GKE customers efficiently obtain accelerators across regions, ensuring workload continuity and optimized resource utilization.

Deploying MCP Servers on GKE: Building AI Agents with ADK and Ray-Served Models

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

This guide shows how to host a Model Context Protocol (MCP) server with Server Sent Events (SSE) transport on Google Kubernetes Engine (GKE). MCP is an open protocol that standardizes how AI agents interact with their environment and external data sources. MCP clients can communicate with the MCP servers using two distinct transport mechanisms:

DWS

Mon, 01 Jan 0001 00:00:00 +0000

The repository contains examples on how to use DWS in GKE. More information about DWS is available here.

Setup and Usage

Prerequisites

Google Cloud account set up.
gcloud command line tool installed and configured to use your GCP project.
kubectl command line utility is installed.
terraform command line installed.

Check out the necessary code files:

git clone https://github.com/ai-on-gke/tutorials-and-examples.git
cd tutorials-and-examples/workflow-orchestration/dws-example

Create Clusters

terraform -chdir=tf init
terraform -chdir=tf plan
terraform -chdir=tf apply -var project_id=<YOUR PROJECT ID>

Install Kueue

VERSION=v0.12.0
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml

Create Kueue resources

kubectl apply -f dws-queues.yaml

Validate installation

Verify the Kueue installation in your GKE cluster

Fine-tuning Gemma 3-1B-it on L4

Mon, 01 Jan 0001 00:00:00 +0000

We’ll walk through fine-tuning a Gemma 3-1B-it model using GKE using L4 GPU. L4 GPU are suitable for many use cases beyond serving models. We will demonstrate how the L4 GPU is a great option for fine tuning LLMs such as Gemma 3, at a fraction of the cost of using a higher end GPU.

Let’s get started and fine-tune Gemma 3-1B-it on the b-mc2/sql-create-context dataset using GKE. Parameter Efficient Fine Tuning (PEFT) and LoRA is used so fine-tuning is posible on GPUs with less GPU memory.

Running Flyte on GKE

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This guide illustrates the deployment of Flyte on Google Kubernetes Engine (GKE) using Helm, utilizing Google Cloud Storage for scalable data storage and Cloud SQL PostgreSQL for a reliable metadata store. By the end of this tutorial, you will have a fully functional Flyte instance on GKE, offering businesses seamless integration with the GCP ecosystem, improved resource efficiency, and cost-effectiveness. ML platform engineers will benefit from the simplified deployment and management provided by GKE, along with the enhanced security achieved by enabling authentication using Google OAuth.

Hugging Face TGI

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This guide demonstrates how to deploy a Hugging Face Text Generation Inference (TGI) server on Google Kubernetes Engine (GKE) using NVIDIA L4 GPUs, enabling you to serve large language models like Mistral-7b-instruct. It walks you through creating a GKE cluster, deploying the TGI application, sending prompts to the model, and monitoring the service’s performance using metrics, while also providing instructions for cleaning up the cluster.

Load Hugging Face Models into Cloud Storage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This guide uses a Kubernetes Job to load meta-llama/Meta-Llama-3-8B model weights hosted on Hugging Face, into a Cloud Storage Bucket, which is used in a vLLM model deployment.

Jupyter on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This repository contains a Terraform template for running JupyterHub on Google Kubernetes Engine.

This module deploys the following resources, once per user:

JupyterHub deployment
User namespace
Kubernetes service accounts

Prerequisites

GCP Project with following APIs enabled

Kserve

Mon, 01 Jan 0001 00:00:00 +0000

KServe is a highly scalable, standards-based platform for model inference on Kubernetes. Installing KServe on GKE Autopilot can be challenging due to the security policies enforced by Autopilot. This tutorial will guide you step by step through the process of installing KServe in a GKE Autopilot cluster.

Additionally, this tutorial includes an example of serving Gemma2 with vLLM in KServe, demonstrating how to utilize GPU resources in KServe on Google Kubernetes Engine (GKE).

Deploying a Persistent Chatbot on Google Cloud Platform with LangChain, Streamlit, and IAP

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

In this tutorial, you will learn how to deploy a chatbot application using LangChain and Streamlit on Google Cloud Platform (GCP).

Llamaindex in GKE cluster

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This tutorial will guide you through creating a robust Retrieval-Augmented Generation (RAG) system using LlamaIndex and deploying it on Google Kubernetes Engine (GKE).

Fine-Tuning Gemma 2-9B on GKE using Metaflow and Argo Workflows

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial will provide instructions on how to deploy and use the Metaflow framework on GKE (Google Kubernetes Engine) and operate AI/ML workloads using Argo-Workflows. The tutorial is designed for ML Platform engineers who plan to use Metaflow for ML workloads on top of GKE by offloading resource-intensive tasks to a managed cluster.

Overview

Provision GKE cluster and install Argo Workflows using Terraform.
Deploy and configure Metaflow’s Metadata Service.
Set up Metaflow to work with Argo Workflows.
Build a fine-tuning image with Cloud Build.
Fine-tune the model using Metaflow and Argo Workflow and GKE.
Deploy the fine-tuned model on GKE for inference.

Before you begin

Ensure you have a gcp project with billing enabled and enabled the GKE API.

Fine-tune gemma-2-9b and track as an experiment in MLFlow

Mon, 01 Jan 0001 00:00:00 +0000

Data scientists often run a lot of experiments and it’s essential to be sure that the best run won’t be lost in numerous experiments. MLFlow helps to track experiments, datasets, metrics, and other data. GKE clusters can provide a lot of resources on demand.

The tutorial overview

In this tutorial we will fine-tune gemma-2-9b using LoRA as an experiment in MLFlow. We will deploy MLFlow on a GKE cluster and set up MLFlow to store artifacts inside a GCS bucket. In the end, we will deploy a fine-tuned model using KServe.

Models as OCI

Mon, 01 Jan 0001 00:00:00 +0000

Package and Deploy from Hugging Face to Artifact Registry and GKE

This repository contains a Google Cloud Build configuration for building and pushing Docker images of Hugging Face models to Google Artifact Registry.

Multikueue, DWS and GKE Autopilot

Mon, 01 Jan 0001 00:00:00 +0000

This repository provides the files needed to demonstrate how to use MultiKueue with Dynamic Workload Scheduler (DWS) and GKE Autopilot. This setup allows you to run workloads across multiple GKE clusters in different regions, automatically leveraging available GPU resources thanks to DWS.

Prerequisites

Google Cloud account set up.
gcloud command line tool installed and configured to use your GCP project.
kubectl command line utility is installed.
terraform command line installed.

Check out the necessary code files:

git clone https://github.com/ai-on-gke/tutorials-and-examples.git
cd tutorials-and-examples/workflow-orchestration/multikueue-dws

Repository Contents

This repository contains the following files:

NVIDIA NeMo

Mon, 01 Jan 0001 00:00:00 +0000

NeMo Framework on Google Kubernetes Engine (GKE) to train Megatron LM

This repository contains an end-to-end walkthrough of training NVIDIA’s NeMo Megatron Large Language Model (LLM) using NeMo framework on Google Kubernetes Engine.

RAG on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This is a sample to deploy a Retrieval Augmented Generation (RAG) application on GKE.

star

The latest recommended release is branch release-1.1.

What is RAG?

RAG is a popular approach for boosting the accuracy of LLM responses, particularly for domain specific or private data sets.

Ray on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This directory contains examples, guides and best practices for running Ray on Google Kubernetes Engine. Most examples use the ray-on-gke terraform module to install KubeRay and deploy RayCluster resources.

Getting Started

It is highly recommended to use the infrastructure terraform module to create your GKE cluster.

Using TPUs with KubeRay on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This directory contains instructions for running KubeRay with TPUs on Google Kubernetes Engine. This includes instructions for installing the KubeRay TPU webhook to enable multi-host TPU workloads.

Prerequisites

Please follow the official Google Cloud documentation for an introduction to TPUs. In particular, please ensure that your GCP project has sufficient quotas to provision the cluster, see this link for details.

Efficient GPU Resource Management for ML Workloads using SkyPilot, Kueue on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial expands on the SkyPilot Tutorial by leveraging Dynamic Workload Scheduler with the help of an open-source project called Kueue Different from the SkyPilot tutorial, this guide shows how to use SkyPilot with Kueue on GKE to efficiently manage ML workloads with dynamic GPU provisioning.

Overview

This tutorial is designed for ML Platform engineers who plan to use SkyPilot to train or serve LLM models on Google Kubernetes Engine (GKE) while utilizing Dynamic Workload Scheduler (DWS) to acquire GPU resources as they become available. It covers installing Kueue and Skypilot, creating a GKE cluster with queue processing enabled GPU node pools, and deploying and running a LLM model. This setup enhances resource efficiency and reduces cost for ML workloads through dynamic GPU provisioning.

Slurm on Google Kubernetes Engine (GKE)

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

This guide shows you how to deploy Slurm on a Google Kubernetes Engine (GKE) cluster.

Time slicing of GPUs with DRA

Mon, 01 Jan 0001 00:00:00 +0000

Background

This tutorial guides you through how to do device sharing of NVIDIA GPUs with Dynamic Resource Allocation on Google Kubernetes Engine (GKE). This guide covers time slicing, with separate guides covering the other device sharing modes, MIG and Multi-Process Service (MPS).

vLLM GPU/TPU Fungibility

Mon, 01 Jan 0001 00:00:00 +0000

Background

This user guide shows you how to optimize AI inference by configuring Google Kubernetes Engine (GKE) to dynamically scale workloads across TPU & GPUs, which helps you better manage demand fluctuations and capacity constraints. In this example we show you how to prioritize high-performance TPU nodes so you deliver optimal speed and responsiveness to your application while ensuring continuous service under peak demand by seamlessly transitioning workloads with additional TPU and GPU nodes, as needed.

Device sharing of GPUs with DRA using MIG

Mon, 01 Jan 0001 00:00:00 +0000

Background

This tutorial guides you through how to do device sharing of NVIDIA GPUs with Dynamic Resource Allocation on Google Kubernetes Engine (GKE). This guide covers Multi-Instance GPU (MIG), with separate guides covering the other device sharing modes, time slicing and Multi-Process Service (MPS).

Building a Multi-Agent Code Development Flow with Flowise on GKE

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial will provide instructions on how to deploy and use Flowise on GKE (Google Kubernetes Engine) to build and operate AI applications using a low-code/no-code approach.

Overview

This tutorial is designed for developers and platform engineers interested in leveraging Flowise on GKE for building customized LLM flows and AI agents, offloading resource-intensive tasks to a managed cluster. Managed clusters automatically handle the complex infrastructure requirements of AI applications like scaling GPU nodes for model inference, managing variable workloads, allowing developers to focus on building AI solutions rather than managing servers. Flowise is a low-code/no-code platform that enables developers to build and deploy AI applications and multi-agent systems through a visual drag-and-drop interface without extensive programming. As an example for this tutorial, we will create a multi-agent application that acts as a software development team with a software developer and a code reviewer which are coordinated by a supervisor. The example demonstrates Flowise’s core multi-agent coordination capabilities. This showcases how AI agents can collaborate on complex tasks—with a supervisor orchestrating between a code writer and reviewer. For more info, you may want to read the docs.

Deploying n8n with agent and tool workflow

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial will provide instructions on how to deploy and use n8n on GKE (Google Kubernetes Engine) to build and operate AI applications using a low-code/no-code approach.

Overview

This tutorial is designed for developers and platform engineers interested in leveraging n8n on GKE for building and hosting powerful workflow automations. By deploying n8n on a managed GKE cluster, you can offload resource-intensive automation tasks, ensure high availability, and scale your automation capabilities without managing underlying server infrastructure.