feat: add ML Diagnostics module and integration for GKE TPU blueprints by AdarshK15 · Pull Request #5350 · GoogleCloudPlatform/cluster-toolkit

AdarshK15 · 2026-03-12T12:10:20Z

Summary

This PR introduces a new mldiagnostics module to automate the installation and configuration of Google Cloud ML Diagnostics (Diagon++) on GKE clusters. It also integrates this capability into the gke-tpu-v6e and gke-tpu-7x blueprints and adds a new Ansible playbook for gke-tpu-v6e and gke-tpu-7x integration tests.

Problem Statement

Setting up Google Cloud ML Diagnostics (Diagon++) on GKE typically involves multiple manual steps, including provisioning IAM permissions, installing Helm charts, and configuring workload namespaces. Automating this setup within the Cluster Toolkit ensures a repeatable, best-practice deployment for profiling, logging, and monitoring AI/ML workloads.

Changes Made

New Module: Added modules/management/mldiagnostics to install ML Diagnostics components (Injection Webhook and Connection Operator) and automatically label the workload namespace for profiling.
Blueprint Integration: Updated gke-tpu-v6e, gke-tpu-7x and their advanced blueprint examples to support optional ML Diagnostics enablement.
Core Module Enhancements:
- gke-cluster: Added the namespace input to support creating Workload Identity resources in a dedicated user workload namespace.
- kubectl-apply: Added cert_manager installation support and templatized the namespace in the Kueue config to create Kueue LocalQueue in the user namespace.
Testing & Verification:
- Added a local validation guide and sample job under modules/management/mldiagnostics/sample-workload-test.
- Added the validation playbook test-gke-ml-diagnostics.yml to the daily integration test suite for both v6e and 7x blueprints.
- Configured the gke-tpu-v6e integration test to use the default namespace for user_namespace, while the gke-tpu-7x test uses a custom namespace (ai-workloads) to validate both use cases.

Documentation

Detailed deployment walkthroughs added to blueprint documentation:
- GKE TPU v6e README
- GKE TPU 7x README

Usage Example

To enable ML Diagnostics on a GKE deployment, include the new module in your blueprint and route the user_namespace from gke-cluster module:

vars:
  user_namespace: ai-workloads
deployment_groups:
- group: primary
  modules:
  - id: gke-ml-diagnostics
    source: modules/management/mldiagnostics
    use: [gke-tpu-v6e-cluster, workload-manager-install]

gemini-code-assist · 2026-03-12T12:10:49Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Cluster Toolkit by introducing a new blueprint that streamlines the deployment of GKE clusters tailored for high-performance AI/ML tasks using TPU v6e accelerators. It simplifies the complex setup of networking, IAM, and Kubernetes scheduling components, while also pre-integrating Google Cloud ML Diagnostics to ensure workloads are immediately observable and diagnosable. This change aims to reduce operational overhead and accelerate the development and deployment of machine learning applications on Google Cloud.

Highlights

New GKE ML Diagnostics Blueprint: Introduced a new blueprint for deploying Google Kubernetes Engine (GKE) clusters with TPU v6e nodes, specifically optimized for AI/ML workloads.
Integrated ML Diagnostics (Diagon++): Integrated Google Cloud ML Diagnostics (Diagon++) directly into the GKE cluster setup, providing out-of-the-box profiling, logging, and monitoring capabilities.
Automated Infrastructure Provisioning: Automated the provisioning of necessary infrastructure, including dual VPC networks, IAM permissions, and Kubernetes components like JobSet and Kueue for enhanced workload management.
Dedicated ML Diagnostics Terraform Module: Added a dedicated Terraform module for ML Diagnostics, enabling easy installation and configuration of Cert-Manager, the ML Diagnostics injection-webhook, and the connection-operator.
Comprehensive Documentation and Sample Job: Provided comprehensive documentation and a sample job YAML to guide users through deployment, verification, and running sample ML workloads.

Changelog

examples/gke-ml-diagnostics/README.md
- Documented the new GKE ML Diagnostics with TPU v6e blueprint, including setup, verification, and cleanup instructions.
examples/gke-ml-diagnostics/gke-ml-diagnostics-tpu-v6e-deployment.yaml
- Defined a new deployment configuration for the GKE ML Diagnostics blueprint.
examples/gke-ml-diagnostics/gke-ml-diagnostics-tpu-v6e.yaml
- Implemented the core blueprint for GKE ML Diagnostics with TPU v6e, orchestrating network, GKE cluster, node pools, and ML diagnostics components.
examples/gke-ml-diagnostics/kueue-configuration.yaml.tftpl
- Created a Terraform template for Kueue resource flavor and cluster queue configuration for TPU v6e.
examples/gke-ml-diagnostics/ml-sample-job.yaml
- Provided a sample Kubernetes JobSet YAML for running ML workloads on the configured GKE cluster.
modules/management/kubectl-apply/README.md
- Updated the README to reflect the new wait output.
modules/management/kubectl-apply/helm_install/main.tf
- Modified the Helm release resource to conditionally add a dependency anchor based on the presence of dependencies.
modules/management/kubectl-apply/outputs.tf
- Added a wait output to establish explicit dependencies for the kubectl-apply module.
modules/management/mldiagnostics/README.md
- Documented the new ML Diagnostics Terraform module.
modules/management/mldiagnostics/main.tf
- Implemented the ML Diagnostics Terraform module, managing the installation of Cert-Manager, ML Diagnostics webhook, and connection operator.
modules/management/mldiagnostics/outputs.tf
- Defined outputs for the ML Diagnostics module, including installation instructions.
modules/management/mldiagnostics/providers.tf
- Configured kubectl and Helm providers for the ML Diagnostics module.
modules/management/mldiagnostics/variables.tf
- Defined input variables for the ML Diagnostics module.
modules/management/mldiagnostics/versions.tf
- Specified Terraform and provider version constraints for the ML Diagnostics module.
modules/scheduler/gke-cluster/README.md
- Updated the README to include the Kubernetes provider and the new k8s_service_account_namespace input.
modules/scheduler/gke-cluster/main.tf
- Added a Kubernetes namespace resource and updated the workload identity module to support configurable namespaces.
modules/scheduler/gke-cluster/outputs.tf
- Adjusted the output message to dynamically reflect the Kubernetes service account namespace.
modules/scheduler/gke-cluster/variables.tf
- Introduced a new variable k8s_service_account_namespace for specifying the Kubernetes service account namespace.

Activity

The pull request was created by AdarshK15.
The description includes a standard submission checklist, indicating it's a community contribution.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new blueprint for ML Diagnostics on GKE with TPUs. It adds a new example, a new mldiagnostics Terraform module, and supporting changes to the kubectl-apply and gke-cluster modules. The changes are well-structured and the new wait output in kubectl-apply is a good pattern for explicit dependencies. However, I've identified a few issues, primarily in the new mldiagnostics module related to incorrect namespace handling and dependency definitions which could cause deployment failures. I've also found some inconsistencies and a typo in the new example's documentation and sample job. My review includes detailed comments and code suggestions to address these points.

LAVEEN · 2026-03-12T13:30:27Z

Please add a PR description

AdarshK15 · 2026-04-19T15:01:16Z

/gcbrun

LAVEEN

LGTM.
Thanks for addressing review comments. Please make sure to run test-gke-ml-diagostics.yml before merging if not ran before.

SwarnaBharathiMantena

LGTM!

AdarshK15 · 2026-05-12T16:51:58Z

Test Failures:

The following tests failed due to insufficient capacity:
Test PR-test-gke-tpu-v6e-flex failed due to a known issue and is disabled in Daily tests.
Test PR-test-gke-tpu-7x also failed due to insufficient capacity, the previous run was successful, previous build link.

All the above test failures are not related to the code changes in this PR.

GoogleCloudPlatform#5350)

AdarshK15 requested a review from annuay-google March 12, 2026 12:12

gemini-code-assist Bot reviewed Mar 12, 2026

View reviewed changes