Adding beta requirements for EndpointSlice API#1286
Adding beta requirements for EndpointSlice API#1286k8s-ci-robot merged 1 commit intokubernetes:masterfrom
Conversation
|
Welcome @robscott! |
|
/cc @freehan |
6b0a710 to
835500e
Compare
|
/cc @thockin |
| // The object labels may contain the following keys: | ||
| // kubernetes.io/service: the label value indicates the name of the | ||
| // service from which the EndpointSlice is derived from. | ||
| // endpointslice.kubernetes.io/managed-by: the label value represents a |
There was a problem hiding this comment.
We'll need a bit more spec on what value that should take. Labels are pretty limited in what they can contain as values.
Another approach might be to add a per-owner label so the key becomes significant. E.g. endpointslice.kubernetes.io/controller-managed: "true"
This allows multiple controllers to think they own a slice, but I am not sure that will be a problem in practice.
There was a problem hiding this comment.
Yeah it's hard to come up with the right label(s) here. Would an EndpointSlice ever become managed by a different resource? I think that's unlikely, but if it happened it would involve controllers changing labels that weren't managed by them (or like you mentioned, just letting multiple controllers try to manage a slice).
If we were to switch the important information from the value to the label (endpointslice.kubernetes.io/controller-managed: "true"), what would other related labels look like? As an example, I think kube-apiserver would be a value for endpointslice.kubernetes.io/managed-by for the initial EndpointSlice created/managed by the apiserver, would we want a separate label for that?
Assuming this is all to support additional controllers managing EndpointSlices, I think the controller-managed name might become confusing as it might seem to indicate the difference between an EndpointSlice being managed by any controller vs being created manually.
There was a problem hiding this comment.
If I had a 3rd party controller it might be example.com/tims-cool-controller: "true"
re apiserver: whether we use label keys or values, it seems appropriate to denote that the apiserver controls its own endpoints.
By now you should know not to let me name things. endpointslice.kubernetes.io/managed-by-endpointslice-controller: "true" is just as fine. No humans will type this.
My main problem with the simpler proposal is that nothing prevents me from accidentally writing a controller called "cool-controller" and you from doing the exact same.
| - Implement e2e tests that ensure both Endpoints and EndpointSlices are tested. | ||
| - Add support for `endpointslice.kubernetes.io/managed-by` label. | ||
| - Add fqdn addressType. | ||
| - Add support for optional appProtocol field on `EndpointPort`. |
There was a problem hiding this comment.
need to spec this carefully for extension
There was a problem hiding this comment.
Should we validate this to only include a set of allowed protocols?
There was a problem hiding this comment.
I don't see a need. There are an endless number of "application protocols" that we cannot really capture all of them, so we would likely just see users requesting new protocols added to the set constantly, or force to do other hacks to specify protocol as done today.
There was a problem hiding this comment.
My thinking was that either we specify a known list and everything else is "x-whatever" or we use a label-prefix style name (my company.com/foo) or something.
It's ugly wrt conventions around naming (HTTP all caps vs company.com/name all lowercase).
Don't need to solve it here, do need to solve it I'm the API specs. One valid answer is "allow anything" (or a specific character set) and don't validate beyond that.
There was a problem hiding this comment.
My thinking was that either we specify a known list and everything else is "x-whatever" or we use a label-prefix style name (my company.com/foo) or something.
It's ugly wrt conventions around naming (HTTP all caps vs company.com/name all lowercase).
Don't need to solve it here, do need to solve it I'm the API specs. One valid answer is "allow anything" (or a specific character set) and don't validate beyond that.
|
|
||
| - #### Why not use a CRD? | ||
|
|
||
| **1. Protobuf is more efficient** Currently CRDs don't support protobuf. In our |
There was a problem hiding this comment.
@wojtek-t FYI
This is unfortunate. I was hoping we could force this to embrace CRD.
| difference in `StreamWatcher.receive` accounting for all of that. With protobuf | ||
| enabled, that function took 1/5th the time of the JSON implementation. | ||
|
|
||
| **2. Validation is too complex** Validation of addresses relies on addressType, |
There was a problem hiding this comment.
@lavalamp @erictune @apelisse and all other CRD-ish folks.
IMO it should be possible to statically express more validation for CRDs - common types like "IP address" and "DNS subdomain" should get distinct validation and error messages.
Cross-field validation is more complex, but I'd like to make a case for simple forms of this to be statically expressed too. e.g. if("X", IPAddress, DNSDomain)
There was a problem hiding this comment.
There's two angles for the "types like IP addresses" -- we can support more "format" values (used in OpenAPI to express complex built-in validation) in kube-apiserver, and some of this can be validated with regexes like:
// this is not correct, but gets the point across
// +kubebuilder:validation:Pattern=`\d{3}\.\d{3}\.\d{3}\.\d{3}`
type IPAddress string
// this would be nice if we could do it
// +kubebuilder:validation:Format=ip-address
type IPAddress stringAs for dependent cross field validation, IIRC some of that was explicitly rejected, but it might be worth re-evaluating.
cc @sttts as well
There was a problem hiding this comment.
would rather make it easier to write validation webhooks than to try to extend OpenAPI to do cross-field validation.
There was a problem hiding this comment.
cross-field valdiation is decently rare too, from what I've seen. Common validation for IP addresses and DNS names would be useful though.
| recreated with OpenAPI validations, the error messages would not be as helpful | ||
| and we would lose the consistency in messaging from the related resources. | ||
|
|
||
| **3. EndpointSlices are required for the API Server to be accessible** In an |
There was a problem hiding this comment.
Yeah, ok. I'd still like to break this but maybe not yet.
There was a problem hiding this comment.
Using CRD sort of required a certain bootstrap sequence:
- CRD ensured
- controller startup.
For CRD consumers, there is no good way to wait for a CRD to showed up. Most implementations just die and wait for the restart to try again.
There was a problem hiding this comment.
we have to address this in general, anyway
835500e to
a330e1e
Compare
a330e1e to
11e5cce
Compare
|
@thockin @dcbw @caseydavenport Can you take a look at this today? The KEP deadline for v1.17 is the end of the day, and it sounds like this graduation criteria should make it in if we want EndpointSlices to graduate to beta. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: robscott, thockin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
I think that having a robust suite of builtin validations is valuable.
Yes, webhooks are pretty easy, but static will always be easier. In my
mind, something like "IP address" is so commonly useful as to warrant this,
but I admit I don't know exactly what the principle should be.
…On Tue, Nov 5, 2019 at 4:00 PM Solly Ross ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In keps/sig-network/20190603-EndpointSlice-API.md
<#1286 (comment)>
:
> @@ -372,6 +430,32 @@ Based on the data collected from user clusters, vast majority (> 99%) of the k8s
The current Endpoints API only includes a boolean state (Ready vs. NotReady) on individual endpoint. However, according to pod life cycle, there are more states (e.g. Graceful Termination, ContainerReary). In order to represent additional states other than Ready/NotReady, a status structure is included for each endpoint. More condition types can be added in the future without compatibility disruptions. As more conditions are added, different consumer (e.g. different kube-proxy implementations) will have the option to evaluate the additional conditions.
+- #### Why not use a CRD?
+
+**1. Protobuf is more efficient** Currently CRDs don't support protobuf. In our
+testing, a protobuf watch is approximately 5x faster than a JSON watch. We used
+pprof to profile 2 versions of kube-proxy using EndpointSlices and running on 2
+different nodes in a 150 node cluster as it scaled up to 15k endpoints. Over the
+15 minute window, kube-proxy with JSON used 17% more CPU time, with the
+difference in `StreamWatcher.receive` accounting for all of that. With protobuf
+enabled, that function took 1/5th the time of the JSON implementation.
+
+**2. Validation is too complex** Validation of addresses relies on addressType,
cross-field valdiation is decently rare too, from what I've seen. Common
validation for IP addresses and DNS names would be useful though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1286?email_source=notifications&email_token=ABKWAVAPS7XTCKME2PNX7CLQSICLNA5CNFSM4I6JJHLKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCKNJ27Q#discussion_r342859180>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVEYNQZHCR3LGGKU6JTQSICLNANCNFSM4I6JJHLA>
.
|
|
I think "stuff that's pretty common for all the kubernetes components" meets the bill. IP addresses, DNS names, and such appear everywhere in kubernetes and are so common in running any system that it'd be hard to argue (IMO) that they were niche. Things that are used in stable APIs that have simple representations but complex validation (e.g. Quantity) are also go candidates, IMO. |
|
(and yes, I realize that's not a hard policy, but it's probably a decent starting point for "what's fairly easy to agree on) |
This PR updates the EndpointSlice KEP to reflect the work necessary for a beta release in 1.17. A significant portion of the changes represent updates that were made as part of the initial alpha API PR but had not yet been reflected here.
This also includes an upgrade plan for adding a new label to indicate which controller is managing an EndpointSlice. Additionally I've outlined the rationale for not choosing a CRD for EndpointSlices.
Enhancement Issue: #752