According to the heap profile, looks like we have two places that consume the most memory:

1. The `DeepCopyInto` method in XDS ResourceVersionTable

Specifically:

gateway/internal/xds/types/resourceversiontable.go

Line 38 in eeb62c8

func (t *ResourceVersionTable) DeepCopyInto(out *ResourceVersionTable) {

      flat  flat%   sum%        cum   cum%
      196.52MB 24.68% 79.92%   196.52MB 24.68%  reflect.New
   ...
         0     0% 97.46%   253.12MB 31.79%  github.com/envoyproxy/gateway/internal/message.HandleSubscription[go.shape.string,go.shape.*uint8]
         0     0% 97.46%   204.05MB 25.63%  github.com/envoyproxy/gateway/internal/xds/translator/runner.(*Runner).subscribeAndTranslate
         0     0% 97.46%   204.05MB 25.63%  github.com/envoyproxy/gateway/internal/xds/translator/runner.(*Runner).subscribeAndTranslate.func1
         0     0% 97.46%   203.55MB 25.56%  github.com/envoyproxy/gateway/internal/xds/types.(*ResourceVersionTable).DeepCopy
         0     0% 97.46%   203.55MB 25.56%  github.com/envoyproxy/gateway/internal/xds/types.(*ResourceVersionTable).DeepCopyInto
         0     0% 97.46%   222.07MB 27.89%  github.com/telepresenceio/watchable.(*Map[go.shape.string,go.shape.*uint8]).Store
         0     0% 97.46%   222.07MB 27.89%  github.com/telepresenceio/watchable.(*Map[go.shape.string,go.shape.*uint8]).unlockedStore
         0     0% 97.46%   222.07MB 27.89%  github.com/telepresenceio/watchable.DeepCopy[go.shape.*uint8]
         0     0% 97.46%   134.51MB 16.89%  google.golang.org/protobuf/internal/impl.(*MessageInfo).initOneofFieldCoders.func5
         0     0% 97.46%   202.04MB 25.37%  google.golang.org/protobuf/internal/impl.(*MessageInfo).merge
         0     0% 97.46%   202.04MB 25.37%  google.golang.org/protobuf/internal/impl.(*MessageInfo).mergePointer
         0     0% 97.46%   150.01MB 18.84%  google.golang.org/protobuf/internal/impl.mergeMessage
         0     0% 97.46%   201.04MB 25.25%  google.golang.org/protobuf/internal/impl.mergeMessageSlice
         0     0% 97.46%   203.55MB 25.56%  google.golang.org/protobuf/proto.Clone
         0     0% 97.46%   202.54MB 25.44%  google.golang.org/protobuf/proto.mergeOptions.mergeMessage

As you can see, it calls protobuf copy frequently, result in calling reflect.New that takes about 25% of total memory usage.

2. YAML Marshal

      flat  flat%   sum%        cum   cum%
  439.84MB 55.24% 55.24%   449.91MB 56.50%  sigs.k8s.io/yaml/goyaml%2ev2.yaml_emitter_emit
   ...
      11MB  1.38% 93.36%    12.50MB  1.57%  sigs.k8s.io/yaml/goyaml%2ev2.(*decoder).scalar
   10.07MB  1.26% 94.63%    10.07MB  1.26%  sigs.k8s.io/yaml/goyaml%2ev2.yaml_string_write_handler
    7.05MB  0.89% 95.51%    97.07MB 12.19%  sigs.k8s.io/yaml/goyaml%2ev2.(*decoder).sequence

No particular code calls, but this part ends up in consuming almost the half of the total memory.

1. The DeepCopyInto method in XDS ResourceVersionTable

2. YAML Marshal

1. The `DeepCopyInto` method in XDS ResourceVersionTable

Why the memory is not increase linearly when the HTTPRoutes scales up? #3698

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions