-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[WIP] Adding extendibility to QP via QPExtension #13122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Fixes: knative#13118 More to come to address extention configuration
|
|
Welcome @davidhadas! It looks like this is your first PR to knative/serving 🎉 |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: davidhadas The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @davidhadas. Thanks for your PR. I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/ok-to-test |
Codecov Report
@@ Coverage Diff @@
## main #13122 +/- ##
==========================================
- Coverage 86.80% 86.76% -0.05%
==========================================
Files 196 197 +1
Lines 14410 14419 +9
==========================================
+ Hits 12509 12511 +2
- Misses 1607 1614 +7
Partials 294 294
Continue to review full report at Codecov.
|
| go get $p | ||
| done | ||
|
|
||
| cat <<EOT >> queue.go |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not simply do this in a separate Go package, e.g. github.com/my-org/queue-proxy and then have main.go:
package main
import "github.com/IBM/go-security-plugs/qpsecurity"
import "knative.dev/serving/pkg/cmd/queue"
import _ "github.com/IBM/go-security-plugs/plugs/testgate"
import _ "github.com/IBM/workload-security-guard/wsgate"
func main() {
qpExtension := qpsecurity.ConfigFromAnnotationFile("/podinfo/annotations", "rtplugs")
defer qpExtension.Shutdown()
queue.mainWithPlugs(qpExtension)
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's break this down to a couple of items:
-
queue.mainWithPlugs(qpExtension)- moving the current code inmain.goto a separate pkg such as "knative.dev/serving/pkg/cmd/queue" (or a pkg under it) seems like a good option
(if everyone is on board with this move) and does simplify the extendibility option as shown. -
As for
qpExtension := qpsecurity.ConfigFromAnnotationFile("/podinfo/annotations", "rtplugs"):
qpsecurityis a QP specific glue pakage, so we need not have this call, it can read from "/podinfo/annotations" (if that is an agreed QP core path for the annotations) during e.Init(). It can reside in the sandbox, moving forward. It can be renamed qpextender. -
defer qpExtension.Shutdown()- as indicated other comment - is a less attractive option due to flush().
| type QPExtension interface { | ||
| Init(logger *zap.SugaredLogger, ctx context.Context) | ||
| Shutdown() | ||
|
|
||
| // If extension does not require to be added to Transport | ||
| // (e.g. when the extensoin is not active), | ||
| // Transport should return next (never return nil) | ||
| Transport(next http.RoundTripper) (roundTripper http.RoundTripper) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two questions:
-
Do we need this interface at all? Between package-level
initstatic functions and theTransportinterface (which could take actx), it seems like you can cover "anytime" initialization and "when loaded" initialization. Similarly,Shutdowncould just be deferred outsideMainWithPlugsunless you expect theShutdownto need to interact with objects that are cleaned up in adeferonMainWithPlugs(I don't see any such thing). -
Do you need to pass both a
ctxand alogger, since you can get yourloggerfrom thectx? If so, the golang convention is that thecontext.Contextshould go first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
To Interface or not to Interface, that is the question:
If we interface:
(a) we open up for growing extendibility as we move forward, without cluttering the core code.
(b) we are not bound to limit ourselves to a single call to the extension (and maneuvering around it)
(c) in the future, if we allow several extension points, we can allow an extension to use more than one
(d) this requires extensions to align with the interface (once it changes to include additional calls)If we do seek to squeeze into a single extendiability point, we can have:
e.transport = Transport(ctx, logger, transport)
This will work for removinge.Init(ctx, logger), but is this the right way forward?It is a reasonable assumption that other extendibility will be needed as we move forward, and all will need
Init(). For example, we may have an extension that only requiresInit()but notTransport()(Do we really want it to useTransport()as a fakeInit(), making the code unreadable?). Also, we may have an extension (security, tracing, metrics, container freeze, etc. ) requiring additional hooks beyondTransport(). Instead of creating a MainWithThisHook() and MainWIthThatHook() for each one, we can make it generic and all will use the same interface (and use what they need from that interface, skipping the rest).As for the
e.Shutdown(), if we avoid the interface, we can solve it inmain().
But if we keep the interface, having it seems like the right way to go.
(As noted in a different comment, flush complexity is removed by havingShutdown()) -
Do we need to pass a logger?
(2a) Right, the order needs to be Init(ctx, logger) - A fix was sent
(2b) We need: One logger to rule them all.
Extensions need to use the logger controlled and being setup by core.
logger, _ := pkglogging.NewLogger(env.ServingLoggingConfig, env.ServingLoggingLevel)
defer flush(logger)
logger = logger.Named("queueproxy").With(
zap.String(logkey.Key, types.NamespacedName{
Namespace: env.ServingNamespace,
Name: env.ServingRevision,
}.String()),
zap.String(logkey.Pod, env.ServingPod))
- Note that this commit does not address extension configuration (not even which extension is activated and which is not during the current execution of QP). For example, an extension may require knowing information such as:
ServingNamespace,ServingRevision,ServingPodIP,ServingPod,ServingServiceetc. Apart from per-extension-specific configs such as on/off the flag and possibly more. Apart from the question about how this information is being configured and propagated to the container, we should ask if we expect each extension to read/parse the config parameters separately or do we transfer at least a set of configs already parsed by the core to all extensions duringInit()- for examplee.Init(ctx, logger, extconfigs)where extconfigs will includeServingNamespaceet al.
| // initialize QPExtensions | ||
| for _, ext := range extensions { | ||
| ext.Init(logger, ctx) | ||
| defer ext.Shutdown() | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only benefit I see of having Shutdown be part of the interface (rather than being part of the function outside MainWithPlugins) is that it may take advantage of the call to flush() on line 131.
The cost is that every implementation of queue.QPExtension must implement Shutdown. The alternative is to make Flush public, and ask callers which care to run flush again.
A third option is to have MainWithPlugs() return a func() for final cleanup which would return func() {flush(logger)}. This would have the advantage that it also worked with other post-MainWithPlugs code, with the disadvantage that you'd want to call MainWithPlugs as defer MainWithPlugs(myhandlers...)() or the like.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep, since defer flush(logger) is called first, and defer ext.Shutdown() after,
flush() will be called after Shutdown()
This seems like a clean way to go.
psschwei
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a general comment: I think it might be useful to consider doing two different PRs here:
- moving the bulk of
maininto a library (pkg/queue/sharedmain?) - allowing the user to pass in additional Handlers / RoundTrippers / WhatHaveYous via the library main function
IMO each of these is doing something slightly different and not necessarily related, so it probably makes sense to split them out. Plus, it'll help keep the PRs small(er), which makes reviewing easier 😄 ...
|
@davidhadas: PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Fixes knative#13118 Alternative to knative#13122 Proposed Changes Enabling downstream and anyone creating an image of queue proxy to extend queue proxy This extendibility is a step to enable the runtime security proposal here Release Note Refactored queue-proxy binary to more easily support out-of-tree extension.
|
An alternative is #13133 |
…3133) * Enable refactoring queue-proxy binary with out-of-tree extensions Fixes #13118 Alternative to #13122 Proposed Changes Enabling downstream and anyone creating an image of queue proxy to extend queue proxy This extendibility is a step to enable the runtime security proposal here Release Note Refactored queue-proxy binary to more easily support out-of-tree extension. * Removing script as to not delay the merge, can add it later if decided * making Defaults concrete * renamed back to config * use d.Transport directly * an overview comment on the Defaults and Env * added printout in case of error during config
|
closing as alternative was merged |
* Enable refactoring queue-proxy binary with out-of-tree extensions Fixes #13118 Alternative to #13122 Proposed Changes Enabling downstream and anyone creating an image of queue proxy to extend queue proxy This extendibility is a step to enable the runtime security proposal here Release Note Refactored queue-proxy binary to more easily support out-of-tree extension. * Removing script as to not delay the merge, can add it later if decided * making Defaults concrete * renamed back to config * use d.Transport directly * QP Extension Configuration Fixes #13118 Annotation based configuration for activat8ing and configuring QP Extensions Support defaults in deployment config map * update-codegen * typos * checksum after typos :) * fix unit tests for queue using volume annotations * removing defaults, will be added in a seperate pull request to focus the review on annotations first * remove default-annotations leftovers * fix gosec shouting * an overview comment on the Defaults and Env * merge QPOptions last commit * podInfo feature added, volumes and volumeMounts aligned * fix gosec linting * removed debug printout leftovers * PodSpecPodInfo default moved from Allowed to Disabled * mainly changes to const values * reduce diff * removed unused code * rm unused constant
Fixes #13118
Proposed Changes
Release Note