HTTP 502 errors on scale downs, graceful shutdown broken with internal encryption #16162
Description
What version of Knative?
0.21.x
Expected Behavior
With internal encryption enabled
spec:
config:
certmanager:
clusterLocalIssuerRef: |
kind: ClusterIssuer
name: knative-selfsigned-issuer
systemInternalIssuerRef: |
kind: ClusterIssuer
name: knative-selfsigned-issuer
features:
kubernetes.podspec-init-containers: Enabled
network:
cluster-local-domain-tls: Enabled
system-internal-tls: Enabledthe prestop hook should respect it and send requests using HTTPS scheme.
Actual Behavior
The queue-proxy logs show an error:
2025-10-16T13:44:32.745982209Z 2025/10/16 13:44:32 http: TLS handshake error from 10.131.0.2:56672: client sent an HTTP request to an HTTPS server
The problem seems to be that the pre-stop hook that is added by knative to the user-container
lifecycle:
preStop:
httpGet:
path: /wait-for-drain
port: 8022
scheme: HTTPfails, as the queue-proxy admin port is HTTPS, not HTTP.
Presumably, that causes the /wait-for-drain pre-stop hook to fail immediately, the user-container is then immediately terminated (instead of waiting for the draining), which would explain the dial tcp 127.0.0.1:8080: connect: connection refused error in the queue-proxy reverseproxy just after TERM is received.
2025-10-16T13:44:32.745982209Z 2025/10/16 13:44:32 http: TLS handshake error from 10.131.0.2:56672: client sent an HTTP request to an HTTPS server
2025-10-16T13:44:32.746618297Z {"severity":"INFO","timestamp":"2025-10-16T13:44:32.746541915Z","logger":"queueproxy","caller":"sharedmain/main.go:305","message":"Received TERM signal, attempting to gracefully shutdown servers.","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt"}
2025-10-16T13:44:32.746641758Z {"severity":"INFO","timestamp":"2025-10-16T13:44:32.746608833Z","logger":"queueproxy","caller":"sharedmain/main.go:306","message":"Sleeping 30s to allow K8s propagation of non-ready state","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt"}
2025-10-16T13:44:32.750895739Z {"severity":"ERROR","timestamp":"2025-10-16T13:44:32.750768505Z","logger":"queueproxy","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 69\nTCP: inuse 17 orphan 0 tw 22 alloc 367 mem 564\nUDP: inuse 0 mem 512\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt","error":"dial tcp 127.0.0.1:8080: connect: connection refused","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\t/workspace/vendor/knative.dev/pkg/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/lib/golang/src/net/http/httputil/reverseproxy.go:486\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\t/workspace/pkg/queue/request_metric.go:201\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\t/workspace/pkg/queue/handler.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\t/workspace/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\t/workspace/pkg/http/handler/timeout.go:118"}
2025-10-16T13:44:32.751656821Z {"severity":"ERROR","timestamp":"2025-10-16T13:44:32.75157073Z","logger":"queueproxy","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 69\nTCP: inuse 17 orphan 0 tw 22 alloc 367 mem 564\nUDP: inuse 0 mem 512\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt","error":"dial tcp 127.0.0.1:8080: connect: connection refused","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\t/workspace/vendor/knative.dev/pkg/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/lib/golang/src/net/http/httputil/reverseproxy.go:486\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\t/workspace/pkg/queue/request_metric.go:201\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\t/workspace/pkg/queue/handler.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\t/workspace/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\t/workspace/pkg/http/handler/timeout.go:118"}Looks like prestop hook doesn't seem to care about anything encryption related https://github.com/knative/serving/blob/main/pkg/reconciler/revision/resources/deploy.go#L101-L112
Steps to Reproduce the Problem
Deploy Knative Serving with enabled tls, like it is shown in the example. After that the problem can be reproduced on modified knative serving performance benchmark "load-test" (modified to use https requests)