fix: K8s API does not accept unauthorized requests by bouskaJ · Pull Request #2111 · sigstore/fulcio

bouskaJ · 2025-07-22T12:32:51Z

Summary

Release Note

The cluster token is now automatically passed to the authentication header for the kubernetes.default.svc OIDC provider

Hayden-IO · 2025-07-25T02:58:04Z

Hey, thanks for the PR. Do you have more information about this issue? The PR linked in the issue was from almost 10 years ago so I'm uncertain why this is now coming up.

I also am not too familiar with Kubernetes setups, so can you confirm if k8sTokenFile is always present? Is whether or not the token is passed something that a deployer can configure?

codecov · 2025-07-25T03:00:45Z

Codecov Report

❌ Patch coverage is 69.23077% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.84%. Comparing base (cf238ac) to head (aaa8d28).
⚠️ Report is 491 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/config/config.go	69.23%	6 Missing and 6 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2111       +/-   ##
===========================================
- Coverage   57.93%   44.84%   -13.09%     
===========================================
  Files          50       72       +22     
  Lines        3119     4669     +1550     
===========================================
+ Hits         1807     2094      +287     
- Misses       1154     2339     +1185     
- Partials      158      236       +78

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bouskaJ · 2025-07-25T09:26:00Z

Hi @haydentherapper thank you for the review. I can confirm that the token file is always present (as you can see in the Kubernetes doc https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#directly-accessing-the-rest-api).

For more information about this issue, you can read https://www.redhat.com/en/blog/what-you-need-to-know-red-hat-openshift-416
Especially following paragraph is important for this context

Reduce unauthenticated user or group access
Beginning with OpenShift 4.16, there's a limit on the permissions given to the system:anonymous user and system:unauthenticated group. Only two roles are allowed by default:

system:openshift:public-info-viewer: required for OIDC workflows
system:public-info-viewer: read-only access to non-sensitive information about the cluster
This applies to new clusters only. Clusters that are upgraded are not affected.

For use cases where anonymous access is needed, the cluster administrator must explicitly add those permissions. For example, if you're using webhooks for BuildConfigs from an external system (like GitHub) that don't support sending auth tokens over HTTP, then you'd need to explicitly add system:webhook permissions. We strongly recommend using local rolebindings instead of clusterrole bindings in such scenarios. A cluster administrator can add back a ClusterRoleBinding for the anonymous user as needed, after evaluating the risks and benefits associated with it

Hayden-IO · 2025-07-25T18:15:57Z

If this is specific to OpenShift, can we make this configurable whether or not to attach a token? I'd like to avoid the risk breaking any existing deployments.

bouskaJ · 2025-07-28T09:13:42Z

@haydentherapper yes, I agree that we should be careful with the backward compatibility. I do not want to add another variable to the https://pkg.go.dev/github.com/sigstore/fulcio/pkg/config#OIDCIssuer that is specific only to Kubernetes issuer, so we have limited options on how to make it configurable.

I added the if file exists check before adding the AUTH header instead. According to the k8s DOC if the file is present, it should always be valid because we follow the documented way on how to connect to the k8s API server. https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#without-using-a-proxy

bouskaJ · 2025-08-06T13:19:00Z

@haydentherapper could you please re-review this PR?

cmurphy

The link to the Kubernetes PR was for a 1.5 release, but it looks like in just the next release they changed its behavior - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#anonymous-requests - so it doesn't seem like that PR is telling the full story.

It looks like Kubernetes supports adding certain endpoints to an AuthenticationConfiguration list to allow anonymous access, could OpenShift consider doing that?

Could access to this endpoint be granted to the system:openshift:public-info-viewer role, which looks like it is already assigned to the anonymous user and is intended for OIDC workflows?

cmurphy · 2025-08-06T20:13:12Z

+	_, verifierPresent := fc.verifiers[k8sIssuerURL]
+	k8sIssuer, issuerConfigured := fc.GetIssuer(k8sIssuerURL)
+	if issuerConfigured && !verifierPresent {
+		// configure static validator for k8s that cover metaIssuers


Can you explain more why the k8s issuer would be present in the config but not already have its provider/verifier created in the loop above? This check doesn't look necessary to me.

The first cycle traverses over static OIDCIssuers. The for cycle covers the following configuration:

OIDCIssuers: map[string]OIDCIssuer{ k8sIssuerURL: { IssuerURL: k8sIssuerURL, ClientID: "sigstore", Type: IssuerTypeKubernetes, }, },

(test https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L866-L873)

For MetaIssuer the GetVerifier function will not get any static verifier without adding it explicitly (see https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config.go#L197). The function would create a new verifier dynamically in that case. This new client will use http.Client without the authorization.

I am creating the additional static verifier in case the k8s issuer can be resolved by https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config.go#L156 but it is not present yet.

This behaviour is tested by https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L875C3-L887C5

If it has issuer URL https://kubernetes.default.svc it will always be an OIDCIssuer, not a MetaIssuer, right?

The config tests don't pass for me when I run them locally so I'm not able to verify the behavior you're describing.

At the very least could you add a comment describing why insertVerifier is being called twice?

cmurphy · 2025-08-06T20:13:32Z

+	return nil
+}
+
+var (


These could all be consts.

Yes, they could, but I make them vars because it allows me to change the value for testing purposes. https://github.com/bouskaJ/fulcio/blob/fix_2110/pkg/config/config_test.go#L905

bouskaJ · 2025-08-07T07:36:03Z

@cmurphy Thank you for your comments.

The link to the Kubernetes PR was for a 1.5 release, but it looks like in just the next release they changed its behavior - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#anonymous-requests - so it doesn't seem like that PR is telling the full story.

yes, I linked the very old PR that might not be relevant to recent changes but the https://www.redhat.com/en/blog/what-you-need-to-know-red-hat-openshift-416 is current and comprehensive source.

It looks like Kubernetes supports adding certain endpoints to an AuthenticationConfiguration list to allow anonymous access, could OpenShift consider doing that?

Could access to this endpoint be granted to the system:openshift:public-info-viewer role, which looks like it is already assigned to the anonymous user and is intended for OIDC workflows?

I agree that both Kubernetes and OpenShift allow for role modifications. However, I don't think it's a good practice to lower the default security on the cluster. We should instead focus on finding a way to work within the cluster's default configuration.

cmurphy · 2025-09-08T18:44:52Z

+	_, verifierPresent := fc.verifiers[k8sIssuerURL]
+	k8sIssuer, issuerConfigured := fc.GetIssuer(k8sIssuerURL)
+	if issuerConfigured && !verifierPresent {
+		// configure static validator for k8s that cover metaIssuers


If it has issuer URL https://kubernetes.default.svc it will always be an OIDCIssuer, not a MetaIssuer, right?

The config tests don't pass for me when I run them locally so I'm not able to verify the behavior you're describing.

At the very least could you add a comment describing why insertVerifier is being called twice?

Signed-off-by: Jan Bouska <jbouska@redhat.com>

gbenhaim · 2026-03-05T09:47:39Z

+	// hence the Gosec G101 warning is a false positive and is suppressed.
+	// #nosec G101
+	k8sTokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+	k8sIssuerURL = "https://kubernetes.default.svc"


is it possible to also support the " https://kubernetes.default.svc.cluster.local" url? cluster created with kubeadm has this url set for the SA issuer.

bouskaJ requested a review from a team as a code owner July 22, 2025 12:32

bouskaJ force-pushed the fix_2110 branch 2 times, most recently from ab11d59 to 4fe495f Compare July 25, 2025 09:05

bouskaJ force-pushed the fix_2110 branch from 4fe495f to 496c291 Compare July 25, 2025 09:30

bouskaJ force-pushed the fix_2110 branch from 496c291 to 2df5d43 Compare July 28, 2025 09:07

bouskaJ force-pushed the fix_2110 branch from 2df5d43 to d143757 Compare July 30, 2025 06:53

cmurphy reviewed Aug 6, 2025

View reviewed changes

bouskaJ force-pushed the fix_2110 branch from d143757 to ef3c423 Compare August 7, 2025 08:14

bouskaJ requested a review from cmurphy August 25, 2025 10:05

cmurphy reviewed Sep 8, 2025

View reviewed changes

bouskaJ force-pushed the fix_2110 branch from ef3c423 to 7a58455 Compare October 6, 2025 12:07

cmurphy approved these changes Oct 10, 2025

View reviewed changes

bobcallaway reviewed Oct 29, 2025

View reviewed changes

Comment thread pkg/config/config.go Outdated

bobcallaway previously approved these changes Oct 29, 2025

View reviewed changes

bobcallaway reviewed Oct 29, 2025

View reviewed changes

Comment thread pkg/config/config.go

bouskaJ dismissed bobcallaway’s stale review via b6eb32f October 29, 2025 13:57

bouskaJ force-pushed the fix_2110 branch from 7a58455 to b6eb32f Compare October 29, 2025 13:57

fix: K8s API does not accept unauthorized requests

aaa8d28

Signed-off-by: Jan Bouska <jbouska@redhat.com>

bouskaJ force-pushed the fix_2110 branch from b6eb32f to aaa8d28 Compare October 30, 2025 07:11

bobcallaway approved these changes Oct 30, 2025

View reviewed changes

bobcallaway merged commit 8c9fc22 into sigstore:main Oct 30, 2025
13 checks passed

bobcallaway mentioned this pull request Nov 14, 2025

k8s authentication fails due to change in pr 2111 #2206

Closed

gbenhaim reviewed Mar 5, 2026

View reviewed changes

Conversation

bouskaJ commented Jul 22, 2025

Summary

Release Note

Uh oh!

Hayden-IO commented Jul 25, 2025

Uh oh!

codecov Bot commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bouskaJ commented Jul 25, 2025

Uh oh!

Hayden-IO commented Jul 25, 2025

Uh oh!

bouskaJ commented Jul 28, 2025

Uh oh!

bouskaJ commented Aug 6, 2025

Uh oh!

cmurphy left a comment

Choose a reason for hiding this comment

Uh oh!

cmurphy Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

bouskaJ Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

cmurphy Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmurphy Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

bouskaJ Aug 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bouskaJ commented Aug 7, 2025

Uh oh!

cmurphy Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbenhaim Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Jul 25, 2025 •

edited

Loading

cmurphy Sep 8, 2025 •

edited

Loading

cmurphy Sep 8, 2025 •

edited

Loading