Clean up custom spot VM variables during standard fallback by rahimkhan19 · Pull Request #5697 · GoogleCloudPlatform/cluster-toolkit

rahimkhan19 · 2026-05-22T11:31:52Z

This CL addresses a bug in the integration tests where the Standard fallback mechanism (which runs standard on-demand VMs if Spot capacity is unavailable) would fail.

Specific spot-related variables passed under cli_deployment_vars (like a3_enable_spot_vm: true) were left untouched in the variables files during standard fallback. This caused the compute nodes to still request Spot instances in zones where Spot capacity was exhausted, resulting in Resource exhausted deployment failures.

We now clean up or update all custom Spot VM variables when falling back to standard VM provisioning.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

Fork your PR branch from the Toolkit "develop" branch (not main)
Test all changes with pre-commit in a local branch #
Confirm that "make tests" passes all tests
Add or modify unit tests to cover code changes
Ensure that unit test coverage remains above 80%
Update all applicable documentation
Follow Cluster Toolkit Contribution guidelines #

…aily test provisioning

gemini-code-assist · 2026-05-22T11:32:02Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the reliability of the integration test suite by ensuring that when the system falls back from Spot to on-demand VM provisioning, all relevant configuration variables are properly sanitized. By removing these stale flags, the deployment process avoids requesting unavailable Spot capacity, preventing resource exhaustion errors during standard fallback scenarios.

Highlights

Bug Fix in Integration Tests: Resolved an issue where standard fallback provisioning failed due to lingering custom Spot VM variables in the configuration files.
Cleanup Logic Enhancement: Updated multiple daily test build scripts to explicitly remove specific Spot VM flags (e.g., a3_enable_spot_vm, g4_enable_spot_vm) when falling back to on-demand instances.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request updates several daily test build configurations to ensure machine-specific spot VM variables are removed from the variables file when spot provisioning is disabled. The reviewer identified that the sed commands used for this cleanup are fragile because they rely on exact string matching. It is recommended to use a more flexible regular expression to handle whitespace variations and a generic pattern to cover all *_enable_spot_vm variables, which would enhance the maintainability and robustness of the scripts.

LAVEEN · 2026-05-26T09:07:05Z

TPUv6e and TPU7x also supports standard provisioning model.

Please check and add if we need a fallback login on below tests :

tools/cloud-build/daily-tests/builds/gke-tpu-7x.yaml
tools/cloud-build/daily-tests/builds/gke-tpu-v6e.yaml

…election and standard provisioning models

rahimkhan19 · 2026-05-26T11:30:41Z

TPUv6e and TPU7x also supports standard provisioning model.

Please check and add if we need a fallback login on below tests :

tools/cloud-build/daily-tests/builds/gke-tpu-7x.yaml

tools/cloud-build/daily-tests/builds/gke-tpu-v6e.yaml

gke-tpu-v6e: The fallback logic is already implemented. It successfully detects a standard fallback, sets ENABLE_SPOT="false", and cleans up the spot-related fields in gke-tpu-v6e.yml.

gke-tpu-7x: Added the standard provisioning fallback and zone lookup to match v6e exactly. The script now:
Sources find_available_zone.sh to dynamically select a zone with TPU v7x capacity (using MACHINE_TYPE=tpu, ACCELERATOR_TYPE=tpu7x-16, TPU_RUNTIME_VERSION=tpu7x-ubuntu-2404).
Dynamically resolves $REGION and $ZONE.
Sets ENABLE_SPOT="false" and strips instance_labels / enable_spot: true from gke-tpu-7x.yml if the provisioning model falls back to STANDARD.

…c zone selection and standard provisioning models" This reverts commit c07d0fe.

rahimkhan19 · 2026-05-26T12:45:59Z

TPUv6e and TPU7x also supports standard provisioning model.
Please check and add if we need a fallback login on below tests :

tools/cloud-build/daily-tests/builds/gke-tpu-7x.yaml

tools/cloud-build/daily-tests/builds/gke-tpu-v6e.yaml

gke-tpu-v6e: The fallback logic is already implemented. It successfully detects a standard fallback, sets ENABLE_SPOT="false", and cleans up the spot-related fields in gke-tpu-v6e.yml.

gke-tpu-7x: Added the standard provisioning fallback and zone lookup to match v6e exactly. The script now: Sources find_available_zone.sh to dynamically select a zone with TPU v7x capacity (using MACHINE_TYPE=tpu, ACCELERATOR_TYPE=tpu7x-16, TPU_RUNTIME_VERSION=tpu7x-ubuntu-2404). Dynamically resolves $REGION and $ZONE. Sets ENABLE_SPOT="false" and strips instance_labels / enable_spot: true from gke-tpu-7x.yml if the provisioning model falls back to STANDARD.

Reverted the change for tpu 7x as For TPU 7x (Only GKE Quota Exists)
Our project has TPU 7x quota allocated exclusively to GKE node pools, and exactly 0 quota for standalone TPU 7x VMs.
When find_available_zone.sh tries to check capacity by running gcloud compute tpus tpu-vm create --accelerator-type=tpu7x-16, the Cloud TPU API instantly rejects the call before it can even check capacity, throwing:
Quota 'TPU7xPerProjectPerZoneForTPUAPI' exhausted. Limit 0 in zone us-central1-c.
However, when GKE actually deploys the blueprint node pool, it successfully provisions because GKE uses the GKE TPU quota instead of the standalone TPU VM quota.
Since the standalone Cloud TPU VM API capacity check is blocked by the zero-quota limit, the find_available_zone.sh script is unable to evaluate TPU 7x capacity.

fix: remove instance-specific spot enablement flags during non-spot d…

1c7d983

…aily test provisioning

gemini-code-assist Bot reviewed May 22, 2026

View reviewed changes

rahimkhan19 added the release-bugfix Added to release notes under the "Bug fixes" heading. label May 22, 2026

rahimkhan19 marked this pull request as ready for review May 25, 2026 07:46

rahimkhan19 requested a review from a team as a code owner May 25, 2026 07:46

feat: update GKE TPU 7x build configuration to support dynamic zone s…

c07d0fe

…election and standard provisioning models

Revert "feat: update GKE TPU 7x build configuration to support dynami…

0e0b57c

…c zone selection and standard provisioning models" This reverts commit c07d0fe.

LAVEEN approved these changes May 26, 2026

View reviewed changes

rahimkhan19 merged commit 5f83ce2 into GoogleCloudPlatform:develop May 26, 2026
13 of 85 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up custom spot VM variables during standard fallback#5697

Clean up custom spot VM variables during standard fallback#5697
rahimkhan19 merged 3 commits into
GoogleCloudPlatform:developfrom
rahimkhan19:tests-spot-variables-fix

rahimkhan19 commented May 22, 2026

Uh oh!

gemini-code-assist Bot commented May 22, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LAVEEN commented May 26, 2026

Uh oh!

rahimkhan19 commented May 26, 2026

Uh oh!

rahimkhan19 commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rahimkhan19 commented May 22, 2026

Submission Checklist

Uh oh!

gemini-code-assist Bot commented May 22, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LAVEEN commented May 26, 2026

Uh oh!

rahimkhan19 commented May 26, 2026

Uh oh!

rahimkhan19 commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants