Skip to content

Adding beta requirements for EndpointSlice API#1286

Merged
k8s-ci-robot merged 1 commit intokubernetes:masterfrom
robscott:endpointslice-beta
Oct 15, 2019
Merged

Adding beta requirements for EndpointSlice API#1286
k8s-ci-robot merged 1 commit intokubernetes:masterfrom
robscott:endpointslice-beta

Conversation

@robscott
Copy link
Copy Markdown
Member

@robscott robscott commented Oct 7, 2019

This PR updates the EndpointSlice KEP to reflect the work necessary for a beta release in 1.17. A significant portion of the changes represent updates that were made as part of the initial alpha API PR but had not yet been reflected here.

This also includes an upgrade plan for adding a new label to indicate which controller is managing an EndpointSlice. Additionally I've outlined the rationale for not choosing a CRD for EndpointSlices.

Enhancement Issue: #752

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

Welcome @robscott!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/network Categorizes an issue or PR as relevant to SIG Network. labels Oct 7, 2019
@robscott
Copy link
Copy Markdown
Member Author

robscott commented Oct 7, 2019

/cc @freehan

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 11, 2019
@robscott
Copy link
Copy Markdown
Member Author

/cc @thockin

@k8s-ci-robot k8s-ci-robot requested a review from thockin October 11, 2019 21:14
// The object labels may contain the following keys:
// kubernetes.io/service: the label value indicates the name of the
// service from which the EndpointSlice is derived from.
// endpointslice.kubernetes.io/managed-by: the label value represents a
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need a bit more spec on what value that should take. Labels are pretty limited in what they can contain as values.

Another approach might be to add a per-owner label so the key becomes significant. E.g. endpointslice.kubernetes.io/controller-managed: "true"

This allows multiple controllers to think they own a slice, but I am not sure that will be a problem in practice.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's hard to come up with the right label(s) here. Would an EndpointSlice ever become managed by a different resource? I think that's unlikely, but if it happened it would involve controllers changing labels that weren't managed by them (or like you mentioned, just letting multiple controllers try to manage a slice).

If we were to switch the important information from the value to the label (endpointslice.kubernetes.io/controller-managed: "true"), what would other related labels look like? As an example, I think kube-apiserver would be a value for endpointslice.kubernetes.io/managed-by for the initial EndpointSlice created/managed by the apiserver, would we want a separate label for that?

Assuming this is all to support additional controllers managing EndpointSlices, I think the controller-managed name might become confusing as it might seem to indicate the difference between an EndpointSlice being managed by any controller vs being created manually.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I had a 3rd party controller it might be example.com/tims-cool-controller: "true"

re apiserver: whether we use label keys or values, it seems appropriate to denote that the apiserver controls its own endpoints.

By now you should know not to let me name things. endpointslice.kubernetes.io/managed-by-endpointslice-controller: "true" is just as fine. No humans will type this.

My main problem with the simpler proposal is that nothing prevents me from accidentally writing a controller called "cool-controller" and you from doing the exact same.

- Implement e2e tests that ensure both Endpoints and EndpointSlices are tested.
- Add support for `endpointslice.kubernetes.io/managed-by` label.
- Add fqdn addressType.
- Add support for optional appProtocol field on `EndpointPort`.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to spec this carefully for extension

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we validate this to only include a set of allowed protocols?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a need. There are an endless number of "application protocols" that we cannot really capture all of them, so we would likely just see users requesting new protocols added to the set constantly, or force to do other hacks to specify protocol as done today.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was that either we specify a known list and everything else is "x-whatever" or we use a label-prefix style name (my company.com/foo) or something.

It's ugly wrt conventions around naming (HTTP all caps vs company.com/name all lowercase).

Don't need to solve it here, do need to solve it I'm the API specs. One valid answer is "allow anything" (or a specific character set) and don't validate beyond that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was that either we specify a known list and everything else is "x-whatever" or we use a label-prefix style name (my company.com/foo) or something.

It's ugly wrt conventions around naming (HTTP all caps vs company.com/name all lowercase).

Don't need to solve it here, do need to solve it I'm the API specs. One valid answer is "allow anything" (or a specific character set) and don't validate beyond that.


- #### Why not use a CRD?

**1. Protobuf is more efficient** Currently CRDs don't support protobuf. In our
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wojtek-t FYI

This is unfortunate. I was hoping we could force this to embrace CRD.

difference in `StreamWatcher.receive` accounting for all of that. With protobuf
enabled, that function took 1/5th the time of the JSON implementation.

**2. Validation is too complex** Validation of addresses relies on addressType,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lavalamp @erictune @apelisse and all other CRD-ish folks.

IMO it should be possible to statically express more validation for CRDs - common types like "IP address" and "DNS subdomain" should get distinct validation and error messages.

Cross-field validation is more complex, but I'd like to make a case for simple forms of this to be statically expressed too. e.g. if("X", IPAddress, DNSDomain)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's two angles for the "types like IP addresses" -- we can support more "format" values (used in OpenAPI to express complex built-in validation) in kube-apiserver, and some of this can be validated with regexes like:

// this is not correct, but gets the point across
// +kubebuilder:validation:Pattern=`\d{3}\.\d{3}\.\d{3}\.\d{3}`
type IPAddress string

// this would be nice if we could do it
// +kubebuilder:validation:Format=ip-address
type IPAddress string

As for dependent cross field validation, IIRC some of that was explicitly rejected, but it might be worth re-evaluating.

cc @sttts as well

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thockin

would rather make it easier to write validation webhooks than to try to extend OpenAPI to do cross-field validation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cross-field valdiation is decently rare too, from what I've seen. Common validation for IP addresses and DNS names would be useful though.

recreated with OpenAPI validations, the error messages would not be as helpful
and we would lose the consistency in messaging from the related resources.

**3. EndpointSlices are required for the API Server to be accessible** In an
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, ok. I'd still like to break this but maybe not yet.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using CRD sort of required a certain bootstrap sequence:

  1. CRD ensured
  2. controller startup.

For CRD consumers, there is no good way to wait for a CRD to showed up. Most implementations just die and wait for the restart to try again.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have to address this in general, anyway

@robscott
Copy link
Copy Markdown
Member Author

@thockin @dcbw @caseydavenport Can you take a look at this today? The KEP deadline for v1.17 is the end of the day, and it sounds like this graduation criteria should make it in if we want EndpointSlices to graduate to beta.

@thockin
Copy link
Copy Markdown
Member

thockin commented Oct 15, 2019

/lgtm
/Approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 15, 2019
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: robscott, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2019
@k8s-ci-robot k8s-ci-robot merged commit cab9087 into kubernetes:master Oct 15, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Oct 15, 2019
@thockin
Copy link
Copy Markdown
Member

thockin commented Nov 6, 2019 via email

@DirectXMan12
Copy link
Copy Markdown
Contributor

I think "stuff that's pretty common for all the kubernetes components" meets the bill. IP addresses, DNS names, and such appear everywhere in kubernetes and are so common in running any system that it'd be hard to argue (IMO) that they were niche. Things that are used in stable APIs that have simple representations but complex validation (e.g. Quantity) are also go candidates, IMO.

@DirectXMan12
Copy link
Copy Markdown
Contributor

DirectXMan12 commented Nov 6, 2019

(and yes, I realize that's not a hard policy, but it's probably a decent starting point for "what's fairly easy to agree on)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants