Skip to content

Conversation

@davidhadas
Copy link
Contributor

@davidhadas davidhadas commented Jul 13, 2022

Fixes #13118

Proposed Changes

Release Note

Refactored queue-proxy binary to more easily support out-of-tree extension.

@knative-prow knative-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 13, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jul 13, 2022

CLA Not Signed

@knative-prow knative-prow bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. area/API API objects and controllers area/autoscale area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features labels Jul 13, 2022
@knative-prow
Copy link

knative-prow bot commented Jul 13, 2022

Welcome @davidhadas! It looks like this is your first PR to knative/serving 🎉

@knative-prow knative-prow bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 13, 2022
@knative-prow
Copy link

knative-prow bot commented Jul 13, 2022

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: davidhadas
To complete the pull request process, please assign psschwei after the PR has been reviewed.
You can assign the PR to them by writing /assign @psschwei in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow
Copy link

knative-prow bot commented Jul 13, 2022

Hi @davidhadas. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@evankanderson
Copy link
Member

/ok-to-test

@knative-prow knative-prow bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 13, 2022
@codecov
Copy link

codecov bot commented Jul 13, 2022

Codecov Report

Merging #13122 (66be24d) into main (3ce8ebb) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

❗ Current head 66be24d differs from pull request most recent head 2d6d4dc. Consider uploading reports for the commit 2d6d4dc to get more accurate results

@@            Coverage Diff             @@
##             main   #13122      +/-   ##
==========================================
- Coverage   86.80%   86.76%   -0.05%     
==========================================
  Files         196      197       +1     
  Lines       14410    14419       +9     
==========================================
+ Hits        12509    12511       +2     
- Misses       1607     1614       +7     
  Partials      294      294              
Impacted Files Coverage Δ
cmd/queue/main.go 0.64% <0.00%> (-0.02%) ⬇️
cmd/queue/queue.go 0.00% <0.00%> (ø)
pkg/reconciler/configuration/configuration.go 83.67% <0.00%> (-1.54%) ⬇️
pkg/autoscaler/statserver/server.go 79.25% <0.00%> (+1.48%) ⬆️
pkg/autoscaler/statforwarder/leases.go 73.95% <0.00%> (+1.56%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ce8ebb...2d6d4dc. Read the comment docs.

go get $p
done

cat <<EOT >> queue.go
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not simply do this in a separate Go package, e.g. github.com/my-org/queue-proxy and then have main.go:

package main

import "github.com/IBM/go-security-plugs/qpsecurity"
import "knative.dev/serving/pkg/cmd/queue"

import _ "github.com/IBM/go-security-plugs/plugs/testgate"
import _ "github.com/IBM/workload-security-guard/wsgate"

func main() {
  qpExtension := qpsecurity.ConfigFromAnnotationFile("/podinfo/annotations", "rtplugs")
  defer qpExtension.Shutdown()
  queue.mainWithPlugs(qpExtension)
}

Copy link
Contributor Author

@davidhadas davidhadas Jul 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's break this down to a couple of items:

  1. queue.mainWithPlugs(qpExtension) - moving the current code in main.go to a separate pkg such as "knative.dev/serving/pkg/cmd/queue" (or a pkg under it) seems like a good option
    (if everyone is on board with this move) and does simplify the extendibility option as shown.

  2. As for qpExtension := qpsecurity.ConfigFromAnnotationFile("/podinfo/annotations", "rtplugs"):
    qpsecurity is a QP specific glue pakage, so we need not have this call, it can read from "/podinfo/annotations" (if that is an agreed QP core path for the annotations) during e.Init(). It can reside in the sandbox, moving forward. It can be renamed qpextender.

  3. defer qpExtension.Shutdown() - as indicated other comment - is a less attractive option due to flush().

Comment on lines 29 to 37
type QPExtension interface {
Init(logger *zap.SugaredLogger, ctx context.Context)
Shutdown()

// If extension does not require to be added to Transport
// (e.g. when the extensoin is not active),
// Transport should return next (never return nil)
Transport(next http.RoundTripper) (roundTripper http.RoundTripper)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions:

  1. Do we need this interface at all? Between package-level init static functions and the Transport interface (which could take a ctx), it seems like you can cover "anytime" initialization and "when loaded" initialization. Similarly, Shutdown could just be deferred outside MainWithPlugs unless you expect the Shutdown to need to interact with objects that are cleaned up in a defer on MainWithPlugs (I don't see any such thing).

  2. Do you need to pass both a ctx and a logger, since you can get your logger from the ctx? If so, the golang convention is that the context.Context should go first.

Copy link
Contributor Author

@davidhadas davidhadas Jul 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. To Interface or not to Interface, that is the question:
    If we interface:
    (a) we open up for growing extendibility as we move forward, without cluttering the core code.
    (b) we are not bound to limit ourselves to a single call to the extension (and maneuvering around it)
    (c) in the future, if we allow several extension points, we can allow an extension to use more than one
    (d) this requires extensions to align with the interface (once it changes to include additional calls)

    If we do seek to squeeze into a single extendiability point, we can have:
    e.transport = Transport(ctx, logger, transport)
    This will work for removing e.Init(ctx, logger), but is this the right way forward?

    It is a reasonable assumption that other extendibility will be needed as we move forward, and all will need Init(). For example, we may have an extension that only requires Init() but not Transport() (Do we really want it to use Transport() as a fake Init(), making the code unreadable?). Also, we may have an extension (security, tracing, metrics, container freeze, etc. ) requiring additional hooks beyond Transport(). Instead of creating a MainWithThisHook() and MainWIthThatHook() for each one, we can make it generic and all will use the same interface (and use what they need from that interface, skipping the rest).

    As for the e.Shutdown(), if we avoid the interface, we can solve it in main().
    But if we keep the interface, having it seems like the right way to go.
    (As noted in a different comment, flush complexity is removed by having Shutdown())

  2. Do we need to pass a logger?

(2a) Right, the order needs to be Init(ctx, logger) - A fix was sent

(2b) We need: One logger to rule them all.
Extensions need to use the logger controlled and being setup by core.

logger, _ := pkglogging.NewLogger(env.ServingLoggingConfig, env.ServingLoggingLevel)
defer flush(logger)

logger = logger.Named("queueproxy").With(
	zap.String(logkey.Key, types.NamespacedName{
		Namespace: env.ServingNamespace,
		Name:      env.ServingRevision,
	}.String()),
	zap.String(logkey.Pod, env.ServingPod)) 
  1. Note that this commit does not address extension configuration (not even which extension is activated and which is not during the current execution of QP). For example, an extension may require knowing information such as: ServingNamespace , ServingRevision , ServingPodIP , ServingPod, ServingService etc. Apart from per-extension-specific configs such as on/off the flag and possibly more. Apart from the question about how this information is being configured and propagated to the container, we should ask if we expect each extension to read/parse the config parameters separately or do we transfer at least a set of configs already parsed by the core to all extensions during Init() - for example e.Init(ctx, logger, extconfigs) where extconfigs will include ServingNamespace et al.

Comment on lines 140 to 144
// initialize QPExtensions
for _, ext := range extensions {
ext.Init(logger, ctx)
defer ext.Shutdown()
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only benefit I see of having Shutdown be part of the interface (rather than being part of the function outside MainWithPlugins) is that it may take advantage of the call to flush() on line 131.

The cost is that every implementation of queue.QPExtension must implement Shutdown. The alternative is to make Flush public, and ask callers which care to run flush again.

A third option is to have MainWithPlugs() return a func() for final cleanup which would return func() {flush(logger)}. This would have the advantage that it also worked with other post-MainWithPlugs code, with the disadvantage that you'd want to call MainWithPlugs as defer MainWithPlugs(myhandlers...)() or the like.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, since defer flush(logger) is called first, and defer ext.Shutdown() after,
flush() will be called after Shutdown()

This seems like a clean way to go.

Copy link
Member

@psschwei psschwei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a general comment: I think it might be useful to consider doing two different PRs here:

  1. moving the bulk of main into a library (pkg/queue/sharedmain ?)
  2. allowing the user to pass in additional Handlers / RoundTrippers / WhatHaveYous via the library main function

IMO each of these is doing something slightly different and not necessarily related, so it probably makes sense to split them out. Plus, it'll help keep the PRs small(er), which makes reviewing easier 😄 ...

This was referenced Jul 14, 2022
@knative-prow-robot knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 19, 2022
@knative-prow-robot
Copy link
Contributor

@davidhadas: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

davidhadas added a commit to davidhadas/serving that referenced this pull request Jul 19, 2022
Fixes knative#13118
Alternative to knative#13122
Proposed Changes
Enabling downstream and anyone creating an image of queue proxy to extend queue proxy
This extendibility is a step to enable the runtime security proposal here

Release Note
Refactored queue-proxy binary to more easily support out-of-tree extension.
@davidhadas
Copy link
Contributor Author

An alternative is #13133

knative-prow bot pushed a commit that referenced this pull request Jul 25, 2022
…3133)

* Enable refactoring queue-proxy binary with out-of-tree extensions

Fixes #13118
Alternative to #13122
Proposed Changes
Enabling downstream and anyone creating an image of queue proxy to extend queue proxy
This extendibility is a step to enable the runtime security proposal here

Release Note
Refactored queue-proxy binary to more easily support out-of-tree extension.

* Removing script as to not delay the merge, can add it later if decided

* making Defaults concrete

* renamed back to config

* use d.Transport directly

* an overview comment on the Defaults and Env

* added printout in case of error during config
@dprotaso
Copy link
Member

closing as alternative was merged

@dprotaso dprotaso closed this Jul 26, 2022
knative-prow bot pushed a commit that referenced this pull request Aug 10, 2022
* Enable refactoring queue-proxy binary with out-of-tree extensions

Fixes #13118
Alternative to #13122
Proposed Changes
Enabling downstream and anyone creating an image of queue proxy to extend queue proxy
This extendibility is a step to enable the runtime security proposal here

Release Note
Refactored queue-proxy binary to more easily support out-of-tree extension.

* Removing script as to not delay the merge, can add it later if decided

* making Defaults concrete

* renamed back to config

* use d.Transport directly

* QP Extension Configuration

Fixes #13118

Annotation based configuration for activat8ing and configuring QP Extensions
Support defaults in deployment config map

* update-codegen

* typos

* checksum after typos :)

* fix unit tests for queue using volume annotations

* removing defaults, will be added in a seperate pull request to focus the review on annotations first

* remove default-annotations leftovers

* fix gosec shouting

* an overview comment on the Defaults and Env

* merge QPOptions last commit

* podInfo feature added, volumes and volumeMounts aligned

* fix gosec linting

* removed debug printout leftovers

* PodSpecPodInfo default moved from Allowed to Disabled

* mainly changes to const values

* reduce diff

* removed unused code

* rm unused constant
@davidhadas davidhadas deleted the QPExtension branch December 13, 2022 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/API API objects and controllers area/autoscale area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queue Proxy Extensions

5 participants