Skip to content

Helm runs out of memory parsing large index.yaml files #9931

@pranavmarla

Description

@pranavmarla

I have access to two privately hosted Helm repos that I was recently attempting to pull Helm charts from. For some reason, only for one of the repos, the helm binary would crash due to a lack of memory whenever I tried the following command: helm repo update -- I even tried removing the repo and re-adding it (helm repo add) but I ran into the same issue:
Specifically, helm would fail with this error message: fatal error: runtime: out of memory.
This was for version 3.2.1 of helm.

I then moved to a brand new machine to see if I could replicate this error. On that machine, I installed the latest version of Helm (3.6.2). I got similar behaviour: Only for that one repo, helm would crash, but now the error message was different: Killed

Eventually though, after running the same helm repo add command multiple times, I guess I got lucky and the command completed. In the process though, helm generated a MASSIVE amount of warning messages that looked like this:

index.go:339: skipping loading invalid entry for chart "..." "..." from ...: validation: chart.metadata.version "..." is invalid

This message appears to be generated when Helm checks the chart versions for SemVer compliance, as per this security notice.

Since the repo that kept causing helm to run out of memory is also the one with apparently a massive amount of charts with noncompliant versions, it seems like the Helm validation is inadvertently causing the memory issues -- i.e. since our Helm repo has so many noncompliant charts, helm runs out of memory, and I am unable to even add that repo.

Unfortunately, I do not own/control the repo, so is there any way we can mitigate this performance hit on the Helm side? Perhaps a flag to the helm binary that tells it not to generate a separate warning message for every single noncompliant chart?

ps: The only confusing thing is that I saw this memory error on two different versions of Helm: 3.2.1 and 3.6.2, but the security notice says that the patch was introduced in 3.5.1 -- so, not sure if my diagnosis of the issue is wrong.

Output of helm version:

  • Machine 1:
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}
  • Machine 2:
version.BuildInfo{Version:"v3.6.2", GitCommit:"ee407bdf364942bcb8e8c665f82e15aa28009b71", GitTreeState:"clean", GoVersion:"go1.16.5"}

Output of kubectl version:

  • Machine 1:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
  • Machine 2:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
AKS

More information:
For more information on my setup, the error messages (including stack trace) that I saw, please see my comment below: link

Metadata

Metadata

Assignees

No one assigned

    Labels

    StalebugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions