Integrate CHS to GKE and Slurm A3U and A4 Daily Tests by simrankaurb · Pull Request #5335 · GoogleCloudPlatform/cluster-toolkit

simrankaurb · 2026-03-11T06:10:48Z

This pull request integrates CHS to the existing daily integration test framework for GKE A3 Ultra and A4. The goal is to provide automated GPU diagnostics and network performance validation as part of the post-deployment test suite.

Key Features & Changes:

Non-Blocking Execution: The CHS validation is designed to be informative rather than disruptive. It utilizes an Ansible rescue block to catch failures, allowing the main playbook to continue execution even if health checks do not pass.
New Validation Playbook (test-chs.yml): Implements a full setup and test cycle, including:
- Python environment configuration using pyenv.
- Cloning and installing CHS into a dedicated virtual environment.
- Running the CHS "standard" suite.
Cloud Build Integration: Modified Cloud Build configurations to inject the CHS_REPO repository URL via Secret Manager and pass it to the integration test playbooks.

gemini-code-assist · 2026-03-11T06:11:14Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the Cluster Health Scanner (CHS) into the existing daily integration test framework. The primary goal is to enhance the robustness of GKE cluster validations by adding automated checks for GPU diagnostics and network performance. This involves a new Ansible playbook for CHS setup and execution, along with necessary updates to Cloud Build configurations to orchestrate these new health checks, ensuring the stability and performance of the tested environments.

Highlights

Cluster Health Scanner (CHS) Integration: Integrated the Cluster Health Scanner into the daily integration test suite to perform automated health checks on GKE clusters.
New Ansible Playbook for CHS: Added a dedicated Ansible playbook (test-chs.yml) responsible for setting up the CHS environment, installing the tool, running its test suite, and validating key metrics like DCGM diagnostics and NCCL bandwidth.
Cloud Build Configuration Updates: Modified several Cloud Build configurations to invoke the new CHS tests and securely pass the CHS repository URL via Secret Manager.
Build Check Temporarily Disabled: Commented out a check_for_running_build step in the gke-a4-onspot.yaml Cloud Build configuration.

Changelog

tools/cloud-build/daily-tests/ansible_playbooks/base-integration-test.yml
- Added a new task block to run CHS tests, delegating to localhost.
- Included a rescue block to gracefully handle and ignore CHS test failures, allowing the playbook to continue execution.
tools/cloud-build/daily-tests/ansible_playbooks/test-validation/test-chs.yml
- Created a new Ansible playbook to set up a Python environment using pyenv and install necessary dependencies.
- Implemented tasks to clone the CHS repository and install it into a dedicated Python virtual environment.
- Added steps to configure a bucket for CHS results and run the CHS lite suite with specified scheduler and machine type.
- Included logic to parse the CHS Bill of Health JSON output and validate specific metrics, such as DCGM diagnostic results and NCCL average bus bandwidth.
tools/cloud-build/daily-tests/builds/gke-a3-ultragpu-onspot.yaml
- Modified the ansible-playbook command to pass the chs_repo variable.
- Updated secretEnv and availableSecrets to include CHS_REPO, fetching its value from Secret Manager.
tools/cloud-build/daily-tests/builds/gke-a3-ultragpu.yaml
- Modified the ansible-playbook command to pass the chs_repo variable.
- Updated secretEnv and availableSecrets to include CHS_REPO, fetching its value from Secret Manager.
tools/cloud-build/daily-tests/builds/gke-a4-onspot.yaml
- Commented out the check_for_running_build step.
- Modified the ansible-playbook command to pass the chs_repo variable.
- Updated secretEnv and availableSecrets to include CHS_REPO, fetching its value from Secret Manager.
tools/cloud-build/daily-tests/builds/gke-a4.yaml
- Modified the ansible-playbook command to pass the chs_repo variable.
- Updated secretEnv and availableSecrets to include CHS_REPO, fetching its value from Secret Manager.

Activity

No specific review comments or activity have been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces integration tests for the Cluster Health Scanner (CHS). The changes involve adding a new Ansible playbook to set up and run CHS, and updating several Cloud Build configurations to trigger these tests. My review focuses on improving the new test playbook for better security, reproducibility, and maintainability by addressing hardcoded values and insecure practices. I've also identified some inconsistencies in the Cloud Build configurations and a potentially problematic commented-out step that should be addressed. Several comments were enhanced with references to existing repository rules to ensure consistency and best practices.

simrankaurb · 2026-03-17T09:55:24Z

/gemini review

gemini-code-assist

Code Review

This pull request integrates the Cluster Health Scanner (CHS) into the daily integration tests for GKE and Slurm clusters, which is a valuable addition for automated diagnostics. The implementation introduces a new Ansible playbook for CHS and updates the Cloud Build configurations accordingly.

My review has identified a few issues that need attention:

A critical merge conflict in tools/cloud-build/daily-tests/tests/ml-a3-ultragpu-onspot-slurm.yml that must be resolved.
Several build configuration files contain a hardcoded project ID, which should be replaced with the ${PROJECT_ID} variable for better portability.
The new Ansible playbook test-chs.yml has some areas for improvement regarding security (insecure directory permissions), maintainability (brittle git refspec), and correctness (a mismatch in the NCCL performance threshold compared to the PR description).

Please address these points to ensure the stability, security, and maintainability of the new testing infrastructure.

sarthakag

LGTM
Let's just ensure that we run the relevant PR tests before submitting the PR; Thanks! 😄

…ed in following commit)

sarthakag

Please squash and merge. Thanks!

gemini-code-assist Bot reviewed Mar 11, 2026

View reviewed changes

simrankaurb force-pushed the chs-integ branch 6 times, most recently from 6717e2e to ceca704 Compare March 11, 2026 12:39

simrankaurb changed the title ~~Chs integ~~ Integrate CHS to GKE A3 and A4 Daily Tests Mar 11, 2026

simrankaurb added the release-improvements Added to release notes under the "Improvements" heading. label Mar 11, 2026

simrankaurb assigned simrankaurb and unassigned simrankaurb Mar 11, 2026

simrankaurb requested review from bytetwin and sarthakag March 11, 2026 14:15

simrankaurb changed the title ~~Integrate CHS to GKE A3 and A4 Daily Tests~~ Integrate CHS to GKE and Slurm A3U and A4 Daily Tests Mar 16, 2026

gemini-code-assist Bot reviewed Mar 17, 2026

View reviewed changes

simrankaurb force-pushed the chs-integ branch from 6731c51 to f540345 Compare March 17, 2026 12:03

sarthakag previously approved these changes Mar 20, 2026

View reviewed changes

Comment thread tools/cloud-build/daily-tests/ansible_playbooks/test-validation/test-chs.yml Outdated

simrankaurb force-pushed the chs-integ branch from e5d2176 to 390bf35 Compare March 31, 2026 09:08

simrankaurb dismissed sarthakag’s stale review via 5f558fa March 31, 2026 13:07

sarthakag marked this pull request as ready for review April 1, 2026 08:26

sarthakag requested review from a team and samskillman as code owners April 1, 2026 08:26

sarthakag previously approved these changes Apr 1, 2026

View reviewed changes

bytetwin reviewed Apr 6, 2026

View reviewed changes

Comment thread tools/cloud-build/daily-tests/ansible_playbooks/test-validation/test-chs.yml Outdated

simrankaurb dismissed sarthakag’s stale review via 6c7c7fd April 7, 2026 14:43

siddhartha-quad and others added 3 commits April 7, 2026 14:44

update terraform-provider version to >= 6.9.0, <= 7.25.0

c6d7029

added pfs-managed-lustre-slurm readme

0c5e902

Fix test scheduling constraints (GoogleCloudPlatform#5383)

87e92d2

simrankaurb added 21 commits April 7, 2026 14:47

fix: async dir

2261166

Add chs repo for a3u and a4 slurm

23b452f

Adding slurm cancel and changing host for slurm

edae4fa

Ensure async directory

cc7fd4f

fix: Configure Enroot and print internal logs on failure(to be revers…

6ae0600

…ed in following commit)

fix: set default Enroot paths

ff71501

Cleaning up code

e75f88c

Setting nccl bandwidth to be consistent with CTK

727efdd

Gemini comments

b4d8cef

Pre-commits fix

0416e15

fixing 0777 error

2fdac2c

Network name a4

bab2e0f

Increase timeout- slurm

d3e6ba7

Removing pyenv and post-test metric validation

cf044db

Minor branch change

b22ea0f

Temp deletion

ca1a04b

sinfo debug

fd2f37d

diag: sinfo

8fa740e

Remove diag

c29f031

Increasing timeout

c946e5a

Running standard suite

036c681

simrankaurb force-pushed the chs-integ branch from 6c7c7fd to 036c681 Compare April 7, 2026 14:48

simrankaurb added 2 commits April 7, 2026 14:58

Running standard suite CHS

1b783ab

Adding accept_hostkey: true

fada654

simrankaurb force-pushed the chs-integ branch from 5411c30 to fada654 Compare April 8, 2026 05:00

Increasing timeout

cf0682d

simrankaurb force-pushed the chs-integ branch from 86bad48 to cf0682d Compare April 8, 2026 08:30

Merge branch 'develop' into chs-integ

e1f0b51

sarthakag approved these changes Apr 8, 2026

View reviewed changes

simrankaurb merged commit a7b8c8d into GoogleCloudPlatform:develop Apr 8, 2026
19 of 76 checks passed

Conversation

simrankaurb commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 11, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simrankaurb commented Mar 17, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarthakag left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sarthakag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

simrankaurb commented Mar 11, 2026 •

edited

Loading

sarthakag left a comment •

edited

Loading