Skip to content

Lookup cluster workload identity from instance metadata#4575

Merged
gusfcarvalho merged 2 commits intoexternal-secrets:mainfrom
xinau:xinau/lookup-cluster-id-from-metadata
Mar 23, 2025
Merged

Lookup cluster workload identity from instance metadata#4575
gusfcarvalho merged 2 commits intoexternal-secrets:mainfrom
xinau:xinau/lookup-cluster-id-from-metadata

Conversation

@xinau
Copy link
Copy Markdown
Contributor

@xinau xinau commented Mar 21, 2025

Problem Statement

When defining a SecretStore that uses the gcpsm (Google Secret Manager) provider, and one wants to use use workload identitity, the spec requires the values for ClusterLocation, ClusterName and ClusterProjectID are set. This adds additional configuration to maintain and makes the configuration more complex then required, as these values can be looked up from the metadata server available to all pods inside a GKE cluster.

Related Issue

Fixes #3540

Proposed Changes

This change solves this issue by looking up the project-id, cluster-location and cluster-name from the metadata server when the values for either ClusterProjectID, ClusterLocation or ClusterName aren't provided.

Checklist

  • I have read the contribution guidelines
  • All commits are signed with git commit --signoff
  • My changes have reasonable test coverage
  • All tests pass with make test
  • I ensured my PR is ready for review with make reviewable

@xinau xinau requested a review from a team as a code owner March 21, 2025 23:19
@xinau xinau requested a review from knelasevero March 21, 2025 23:19
@xinau
Copy link
Copy Markdown
Contributor Author

xinau commented Mar 21, 2025

cc @gusfcarvalho

@xinau
Lookup cluster identity from instance metadata
e5af20c 
issues: external-secrets#3540

This change adds the capability to the gcp secret manager provider to
lookup the clusters workload id pool and provider from the metadata
server.

Signed-off-by: Felix Ehrenpfort <felix@ehrenpfort.de>
@xinau xinau force-pushed the xinau/lookup-cluster-id-from-metadata branch from f043b98 to e5af20c Compare March 21, 2025 23:25
@gusfcarvalho
Copy link
Copy Markdown
Member

gusfcarvalho commented Mar 22, 2025

Hi @xinau ! Thanks for your contribution 💪 🥳

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Mar 23, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@gusfcarvalho
Copy link
Copy Markdown
Member

gusfcarvalho commented Mar 23, 2025

/ok-to-test sha=bcfd9183ebd16f61a8e610bc3b34ef45046b84ca

@eso-service-account-app
Copy link
Copy Markdown
Contributor

eso-service-account-app bot commented Mar 23, 2025

[Bot] - ✅ e2e for bcfd9183ebd16f61a8e610bc3b34ef45046b84ca passed

@gusfcarvalho
Copy link
Copy Markdown
Member

gusfcarvalho commented Mar 23, 2025

Thanks for your contribution!! 🚀🚀

@gusfcarvalho gusfcarvalho merged commit b6acfe8 into external-secrets:main Mar 23, 2025
19 of 20 checks passed
ivankatliarchuk added a commit to gofogo/external-secrets-fork that referenced this pull request Mar 25, 2025
@ivankatliarchuk
Merge branch 'main' into feat-aws-tags-1821
a289b42 
* main:
  update dependencies (external-secrets#4589)
  chore(deps): bump fossas/fossa-action from 1.5.0 to 1.6.0 (external-secrets#4586)
  chore(deps): bump mkdocs-material from 9.6.8 to 9.6.9 in /hack/api-docs (external-secrets#4588)
  chore(deps): bump platformdirs from 4.3.6 to 4.3.7 in /hack/api-docs (external-secrets#4587)
  chore(deps): bump github/codeql-action from 3.28.11 to 3.28.12 (external-secrets#4585)
  chore(deps): bump actions/setup-go from 5.3.0 to 5.4.0 (external-secrets#4584)
  chore(deps): bump golangci/golangci-lint-action from 6.5.1 to 6.5.2 (external-secrets#4583)
  chore(deps): bump actions/cache from 4.2.2 to 4.2.3 (external-secrets#4582)
  chore(deps): bump ubi8/ubi from `5993454` to `8bd1b63` (external-secrets#4581)
  Lookup cluster identity from instance metadata (external-secrets#4575)
  adding conjur description (external-secrets#4578)
  fix: bump jwt for cve fix (external-secrets#4580)
  Add `webhook.create: false` warning comment (external-secrets#4579)
@jbehrends jbehrends mentioned this pull request Apr 4, 2025
Closed
@ionicsolutions ionicsolutions mentioned this pull request Aug 27, 2025
Merged
5 tasks
patjlm added a commit to patjlm/external-secrets that referenced this pull request Feb 3, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
1c98b15 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
@patjlm patjlm mentioned this pull request Feb 3, 2026
Merged
5 tasks
patjlm added a commit to patjlm/external-secrets that referenced this pull request Feb 9, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
43811a3 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
patjlm added a commit to patjlm/external-secrets that referenced this pull request Mar 4, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
9bd9b89 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
patjlm added a commit to patjlm/external-secrets that referenced this pull request Mar 6, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
a1240cb 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
patjlm added a commit to patjlm/external-secrets that referenced this pull request Mar 6, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
7293c3b 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
patjlm added a commit to patjlm/external-secrets that referenced this pull request Mar 6, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
a5416cc 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
patjlm added a commit to patjlm/external-secrets that referenced this pull request Mar 11, 2026
@patjlm
feat(gcpsm): auto-detect projectID from GCP metadata server
83ad7d0 
Extends metadata server auto-detection to include the projectID field
when using Workload Identity in GKE. Builds upon the foundation from
external-secrets#4575, external-secrets#4622, and external-secrets#5208 which made cluster identity parameters optional.

Problem:
While cluster identity parameters (ClusterLocation, ClusterName,
ClusterProjectID) became optional via metadata server lookup, the main
projectID field (specifying where secrets are stored) remained required.
This forced users to parameterize SecretStore configurations per-cluster,
preventing static GitOps deployments via Config Sync/Fleet in
multi-project environments.

Solution:
When projectID is omitted from SecretStore/ClusterSecretStore spec:
1. clusterProjectID() falls back to GCP metadata server
2. NewClient() populates gcpStore.ProjectID with detected value

This allows the operator to access secrets in the same project as the
GKE cluster without explicit configuration.

Use Cases:
- Portable SecretStore configs across multiple GCP projects
- Static bootstrap manifests via Config Sync without parameterization
- Eliminates cross-cluster network dependencies for private GKE clusters

Changes:
- provider.go: Add metadata server fallback in clusterProjectID()
- provider.go: Populate gcpStore.ProjectID from clusterProjectID when empty
- workload_identity_test.go: Add TestClusterProjectIDMetadataFallback
- docs: Document projectID auto-detection feature with examples

Note: projectID remains required when using static service account
credentials, running outside GKE, or accessing secrets in a different
project than the GKE cluster.

Signed-off-by: Patrick Martin <patmarti@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gusfcarvalho gusfcarvalho gusfcarvalho approved these changes

@knelasevero knelasevero Awaiting requested review from knelasevero knelasevero was automatically assigned from external-secrets/maintainers

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

gcpsm provider - Would like the ability to auto detect default values for auth.workloadIdentity cluster(Location|Name|Project)

2 participants

@xinau @gusfcarvalho