Problem Summary:

The Issue: When deploying GKE TPU clusters with Pathways enabled (enable_pathways_for_tpus: true), jobs would fail to schedule and remain in a Pending or Suspended state. While the cluster successfully provisioned the necessary cpu-np node pool for Pathways, the Kueue ClusterQueue was only configured to cover google.com/tpu resources, lacking coverage for the cpu and memory requested by the Pathways head pods.

Root Cause: The problem stemmed from missing variable plumbing between blueprint modules and limitations in the ClusterQueue merging logic in the kubectl-apply module:

• Missing Flag Propagation: The enable_pathways_for_tpus flag was set on the gke-cluster module (triggering node pool creation) but was not visible to the kubectl-apply module (responsible for Kueue config), so the internal pathways template with CPU/Memory quotas was never loaded.
• Destructive Merging: Even if loaded, the previous merging logic in kubectl-apply would overwrite user-defined fields in the ClusterQueue (like cohort or namespace selectors) with the pathways template defaults, instead of combining them.
• Name Mismatch: The DWS-specific templates used a different ClusterQueue name (dws-cluster-queue), preventing the auto-merge logic which relied on the standard name cluster-queue.

Solution:To resolve this comprehensively and seamlessly without resorting to hardcoded values in user templates:

• Seamless Variable Plumbing:

  • Declared enable_pathways_for_tpus as a top-level variable in the vars section of the blueprints and passed it to the gke-cluster module.
  • Added output "enable_pathways_for_tpus" to the gke-cluster module and a matching top-level variable "enable_pathways_for_tpus" to the kubectl-apply module.
    This allows Cluster Toolkit to automatically wire the flag from gke-cluster to kubectl-apply via the use mechanism, removing the need to manually pass it in the blueprint for kubectl-apply.

• Robust ClusterQueue Merging:

  • Updated kubectl-apply/main.tf to perform a merge of the spec and metadata maps of matching ClusterQueue resources. This ensures that user-defined fields (like cohort, labels, and annotations) are preserved while still combining the resourceGroups arrays.
  • Added robust null-safety checks to the merging logic to handle cases where metadata or spec might be empty/null in YAML documents.

• Template Alignment:

  • Aligned the DWS template (tpu-dws-queues.yaml.tftpl) to use the standard cluster-queue name, enabling successful auto-merging with the Pathways quotas.

This fix ensures that Pathways quotas are automatically and safely injected into the user's ClusterQueue across all 8 relevant TPU blueprints.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #