Skip to content

exec credential provider: wire in cluster info#91192

Closed
enj wants to merge 3 commits intokubernetes:masterfrom
enj:enj/f/exec_plugin_cluster
Closed

exec credential provider: wire in cluster info#91192
enj wants to merge 3 commits intokubernetes:masterfrom
enj:enj/f/exec_plugin_cluster

Conversation

@enj
Copy link
Copy Markdown
Member

@enj enj commented May 18, 2020

Signed-off-by: Monis Khan mok@vmware.com

/kind feature
/sig auth
/milestone v1.19
/assign @mikedanese
/priority important-soon
@kubernetes/sig-auth-pr-reviews
xref: kubernetes/enhancements#1711

client-go credential plugins are now passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable.
Need to update https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/auth Categorizes an issue or PR as relevant to SIG Auth. kind/feature Categorizes issue or PR as related to a new feature. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 18, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone May 18, 2020
@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 18, 2020
@k8s-ci-robot k8s-ci-robot requested review from deads2k and dims May 18, 2020 06:15
@k8s-ci-robot k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels May 18, 2020
@enj
Copy link
Copy Markdown
Member Author

enj commented May 18, 2020

/retest

@fedebongio
Copy link
Copy Markdown
Contributor

/remove-sig api-machinery

@k8s-ci-robot k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 19, 2020
@bai
Copy link
Copy Markdown

bai commented May 29, 2020

👋 Hello from Bug Triage team! The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As the PR is tagged for 1.19, is it still planned for this release?

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

@bai: GitHub didn't allow me to request PR reviews from the following users: enj.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

👋 Hello from Bug Triage team! The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As the PR is tagged for 1.19, is it still planned for this release?

/cc @enj

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 13, 2020
@enj enj force-pushed the enj/f/exec_plugin_cluster branch from ca531c9 to 4d78dbb Compare June 16, 2020 20:24
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 16, 2020
@enj
Copy link
Copy Markdown
Member Author

enj commented Jun 17, 2020

/retest

@mikedanese
Copy link
Copy Markdown
Member

/lgtm
/approve

Signed-off-by: Monis Khan <mok@vmware.com>
@enj enj force-pushed the enj/f/exec_plugin_cluster branch from a2fdc3d to 17df008 Compare June 24, 2020 02:36
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: enj, mikedanese
To complete the pull request process, please assign liggitt
You can assign the PR to them by writing /assign @liggitt in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

@enj: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-conformance-kind-ipv6-parallel a2fdc3d6a5f25ddcbe57aa2bc373eb4b067f6ef4 link /test pull-kubernetes-conformance-kind-ipv6-parallel

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@enj
Copy link
Copy Markdown
Member Author

enj commented Jun 24, 2020

/retest

@bai
Copy link
Copy Markdown

bai commented Jun 24, 2020

Hey 👋 Bug Triage Shadow here. Looks like this PR is 🍏 and is waiting for final reviews 👀 .

Wanted to post a friendly reminder that code freeze starts tomorrow to make sure this one doesn't fall through the cracks 🙏

@liggitt
Copy link
Copy Markdown
Member

liggitt commented Jun 24, 2020

code freeze starts tomorrow

code freeze has been moved to July 9th per https://github.com/kubernetes/sig-release/tree/master/releases/release-1.19#tldr

if c.ExecProvider != nil {
provider, err := exec.GetAuthenticator(c.ExecProvider)
if c.Exec.ExecProvider != nil {
caData, err := dataFromSliceOrFile(c.CAData, c.CAFile)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for a CAFile config, this reads/freezes the data from the file at TransportConfig() construction time rather than transport construction time... probably not a big deal, but notable... could end up with different content here than is used in the transport itself.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's also maybe slightly unexpected that TransportConfig() would do file reads...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could pass in the logic as a closure to delay the computation if you think the extra complexity would be worth it?

Comment on lines +84 to +85
// Cluster contains information to allow an exec plugin to communicate
// with the kubernetes cluster being authenticated to.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed some of the discussion around this, so asked a question at https://github.com/kubernetes/enhancements/pull/1711/files#r445956764, echoing here:

I can see use cases for cluster hostname and custom config bits, but I was surprised by the ServerName and CAData bits... are we expecting exec plugins to call out to the server we are asking them to provide credentials for?

If so, don't we need to include ProxyURL and InsecureSkipTLSVerify?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason not to just use the serialized version of the cluster stanza from the kubeconfig file more or less as-is?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was surprised by the ServerName and CAData bits... are we expecting exec plugins to call out to the server we are asking them to provide credentials for?

Yes, similar to what oc login does today to determine the location of the OAuth server (i.e. read this well known config map that tells you something dynamic about the cluster).

If so, don't we need to include ProxyURL and InsecureSkipTLSVerify?

The per-cluster ProxyURL was added in v1.19 without me noticing, but yeah, we should include that. I left out InsecureSkipTLSVerify on purpose to discourage its use...

is there a reason not to just use the serialized version of the cluster stanza from the kubeconfig file more or less as-is?

I had that in the original draft of the KEP updates. See kubernetes/enhancements#1711 (comment). @mikedanese leaned towards a new struct, and I liked the idea of just having a single well-known extension key to pass through instead of just dumping everything.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened a new PR into the KEP to try to address this design update: kubernetes/enhancements#1927.

// cluster:
// ...
// extensions:
// - name: exec # reserved extension name for per cluster exec config
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's ok to declare an unqualified name (no kubernetes.io/ prefix) reserved after the fact like this.

would it be better for the exec config to opt into getting cluster info (and maybe even identify the named cluster extension it wants)?

clusters:
- name: my-cluster
  cluster:
    ...
    extensions:
    - name: myexec
      extension:
        audience: 06e3fbd18de8  # arbitrary config
users:
- name: myuser
  user:
    exec:
      ...
      includeClusterConfig: true
      includeClusterExtension: myexec

that would:

  1. avoid having a magic extension name
  2. avoid clusters only working with a single plugin
  3. avoid sending exec plugins data they did not expect/want (CA data specifically could be quite large to include in the env)
  4. avoid multiple calls to an exec plugin for different clusters when it didn't care about the cluster and the already-fetched credential could have been used

Copy link
Copy Markdown
Member Author

@enj enj Jun 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would personally lean towards just changing exec to client.authentication.k8s.io/exec to prevent any possible collision instead of expanding ExecConfig (though I suspect that we would be fine using exec).

avoid having a magic extension name

I am not sure this is much different from the exec: stanza under user:.

avoid clusters only working with a single plugin

Somehow I do not see a user having the same cluster with multiple distinct exec plugins configured. I realize it is technically possible, but not exactly practical or common.

avoid sending exec plugins data they did not expect/want (CA data specifically could be quite large to include in the env)

I think we can handle the extra 2 KB via the env var...

avoid multiple calls to an exec plugin for different clusters when it didn't care about the cluster and the already-fetched credential could have been used

Using the same credential across clusters is an incredibly unsafe pattern. If you are using a bearer token, you are at risk of a compromised cluster replaying your credentials. For both tokens and certs, if the credential is stolen from disk, it can be used against multiple clusters. Combine that with no revocation of certs, and you are really asking for trouble. I am okay adding the latency of exec-ing the binary per cluster as the binary will likely need to do some form of caching itself to support clients like kubectl.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened a new PR into the KEP to try to address this design update: kubernetes/enhancements#1927.

Comment on lines +84 to +85
// Cluster contains information to allow an exec plugin to communicate
// with the kubernetes cluster being authenticated to.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason not to just use the serialized version of the cluster stanza from the kubeconfig file more or less as-is?

conf: conf,
cluster: cluster,
}
return spewConfig.Sprint(key)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this do the right thing printing the runtime.Config object?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the new test cases in TestCacheKey should cover that.

@liggitt liggitt modified the milestones: v1.19, v1.20 Jul 17, 2020
@dims
Copy link
Copy Markdown
Member

dims commented Jul 27, 2020

/uncc

@k8s-ci-robot k8s-ci-robot removed the request for review from dims July 27, 2020 13:06
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

@enj: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sayanchowdhury
Copy link
Copy Markdown
Member

Hi @enj! 👋🏽 I'm from the Bug Triage Team! The code freeze for 1.20 release is starting November 12th which is about 3 weeks from now. As the PR is tagged for 1.20, is it still planned for this release?

@ankeesler
Copy link
Copy Markdown
Contributor

Hey @sayanchowdhury - we have opened a new PR which has a superset of the stuff included in this PR. Here is the new PR: #95489.

I opened the new PR since I wanted to add right on top of @enj's work. I think if that superset PR gets merged, we can close this PR since @enj's work here will be included in the other PR.

@sayanchowdhury
Copy link
Copy Markdown
Member

Thanks @ankeesler for the update 👍🏽

@dims
Copy link
Copy Markdown
Member

dims commented Oct 22, 2020

subsumed by #95489

/close

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

@dims: Closed this PR.

Details

In response to this:

subsumed by #95489

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added a commit that referenced this pull request Oct 30, 2020
…cluster

exec credential provider: wire in cluster info (superset of #91192)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/auth Categorizes an issue or PR as relevant to SIG Auth. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants