Skip to content

Controller leaks Vault token leases #376

@cooperbenson-qz

Description

@cooperbenson-qz

Describe the solution you'd like
There are a few potential solutions (ordered by personal preference):

  1. Cache the clients created for the SecretStores
  2. Revoke the acquired token as a part of the Close method

What is the added value?
Right now, the controller creates a new Vault token on each reconcile call if they aren't using token auth (which is less secure than the other options). By default, Vault auth backends will issue a service token with a TTL of 768 hours (32 days). As long as a service token has not been revoked or expired, Vault will maintain a lease for that token in the backend. This resulted in the number of leases in our Vault cluster eventually building up to the point that the cluster was unable to function anymore. Before #169 was fixed this took our cluster completely down in the space of a few days.

This could be worked around by using a short-lived service token or batch token, but this would be counter-intuitive for most users since the controller is a long-running service and require users to configure their Vault in a particular way to support this tool.

Give us examples of the outcome

  1. The Vault provider client is cached between reconciliations and handles renewing the token automatically with something like the provided LifetimeWatcher
  2. The controller makes a call to revoke the token when it's done

Observations (Constraints, Context, etc):
N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions