Why we need this change ?

When managing Kubernetes manifests with multiple blueprints using the kubectl-apply module, completely removing a kubectl-apply block from a blueprint results in a "Error: Provider configuration not present" error.

  - id: workload_manager_install
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      kueue:
        install: true
      jobset:
        install: true

  - id: workload_manager_config
    source: modules/management/kubectl-apply
    use: [gke_cluster]
    settings:
      apply_manifests:
      - source: $(ghpc_stage("maxtext-gke-a3-files"))/config-map.yaml.tftpl
        template_vars: {name: "a3plus-benchmark-resources-configmap", num_nodes: "1"}
      - source: $(ghpc_stage("maxtext-gke-a3-files"))/kueue-credentials.yaml.tftpl
        template_vars: {num_chips: "8"}

Similar issue is observed for PersistentVolume and PersistentVolumeClaims, when provisioning these resources using gke-persistent-volume module, which internally uses kubectl-apply module for k8s manifest deployment.

  - id: data-bucket-pv
    source: modules/file-system/gke-persistent-volume
    use: [gke_cluster, data-bucket]
    settings: {capacity_gb: 5000}
    
 - id: shared-filestore-pv
    source: modules/file-system/gke-persistent-volume
    use: [gke_cluster, filestore]

Removing workload_manager_config or persistent_volume and recreating and redeploying gives us the error:

Error: Provider configuration not present

To work with
module.workload_manager_config.module.kubectl_apply_manifests["1"].kubectl_manifest.apply_doc["2"]
(orphan) its original provider configuration at
module.workload_manager_config.provider["registry.terraform.io/gavinbunney/kubectl"]
is required, but it has been removed. This occurs when a provider
configuration is removed while objects created by that provider still exist
in the state. Re-add the provider configuration to destroy
module.workload_manager_config.module.kubectl_apply_manifests["1"].kubectl_manifest.apply_doc["2"]
(orphan), after which you can remove the provider configuration again.

RCA

The issue stems from how the provider is defined. Unlike other Terraform providers which are typically defined at the root level, the kubectl-apply provider is defined within the modules/management/kubectl-apply/providers.tf file inside the module itself.

This becomes problematic when deleting a blueprint configuration that uses this module. While the module's folder remains, Terraform effectively loses track of it and its associated provider. Consequently, Terraform can't determine the correct provider (kubectl_apply_manifests) to use when attempting to delete the resources associated with the removed module.

This behavior is due to Terraform's handling of child modules and their providers. You can find a more detailed explanation in this Stack Overflow answer: https://stackoverflow.com/a/58403262

Solution

Moving kubectl provider to the root module (under TerraformProviders block) of the blueprint group, if the group contains dependent module of gke-cluster or pre-existing-gke-cluster module.

Backward Compatibility Test

Tested this out for example blueprint (storage-gke.yaml, gke-a3-highgpu.yaml), following are the steps.

• on develop branch, perform gcluster deploy for example blueprint.
• switched over to feature branch and performed make to update gcluster binary.
• performed gcluster deploy, got a warning of past deployment was performed on diff gcluster version, suggesting to use --force based on user discretion.

Creating deployment folder "gke-a3-highgpu" ...
Error: ghpc_version has changed from "45068640-dirty" to "90aa74bb-dirty", using different versions of GHPC to update a live deployment is not officially supported
Hint: Use `--force` to overwrite the deployment anyway. Proceed at your own risk.

• used --force flag and did the deployment using feature branch and it worked as expected.
• Only recreate of kubectl-apply module related resource were required.
• Scenario of workload_manager_config and persistent_volume module removal and recreate worked as expected.

How did we test this out ?

• Use the same setup as in issue description. Remove the workload_manager_config module and recreate, redeploy. Verify that the error does not occur and infra is removed as expected
• Use the same setup as in issue description. Remove the persistent_volume module and recreate, redeploy. Verify that the error does not occur and infra is removed as expected
• Test for backward compatibility
• PR cloud build for all GKE related blueprint should have OK status

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #