Very high memory usage after increased churn of HTTPRoute - Memory leak(?) #6919
Description
Description:
We have somewhat recently introduced preview environments in our setup. This means a lot more churn creating and deleting HTTPRoutes. Since this introduction we have observed worse and worse memory usage from envoy-gateway(The controllers).
Before the change: <512MB memory usage per controller
Now: >=16GB per controller when they OOM.
Repro steps:
We create and delete 22 HTTPRoutes per beta environment which we currently have 50 of + a bunch of other HTTPRoutes.
Let me know what other information is required to debug this.
Environment:
Note that we upgraded from 1.3.0 to 1.5.0 on the 28th.
❯ kubectl version
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.6-gke.1060000
❯ egctl version
client: v1.5.0
server:
- Name: envoy-gateway-56f7549fb-52l8m
Namespace: envoy-gateway-system
envoyGatewayVersion: v1.5.0
envoyProxyVersion: distroless-v1.35.0
gatewayAPIVersion: v1.3.1-0.20250527223622-54df0a899c1c
gitCommitID: c2e5b2e118a4d81db698d22d4ff4ebb0211ec8a2
golangVersion: go1.24.6
- Name: envoy-gateway-56f7549fb-thwqw
Namespace: envoy-gateway-system
envoyGatewayVersion: v1.5.0
envoyProxyVersion: distroless-v1.35.0
gatewayAPIVersion: v1.3.1-0.20250527223622-54df0a899c1c
gitCommitID: c2e5b2e118a4d81db698d22d4ff4ebb0211ec8a2
golangVersion: go1.24.6
- Name: envoy-gateway-56f7549fb-zk45c
Namespace: envoy-gateway-system
envoyGatewayVersion: v1.5.0
envoyProxyVersion: distroless-v1.35.0
gatewayAPIVersion: v1.3.1-0.20250527223622-54df0a899c1c
gitCommitID: c2e5b2e118a4d81db698d22d4ff4ebb0211ec8a2
golangVersion: go1.24.6
This is memory usage from a couple of days before we introduced preview environments(25th at 07:00)
entire workload dashboard for the same time range
Envoy gateway global dashboard for the same time range
