Proposal

The prometheus right now support read password from file
But besides read from file, we want to also be able to read from the other password provider for example AWS secret manager, Azure Key Vault provider etc.

We do have the secrets store CSI driver (https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html) to mount the secrets to pod so that we can use the receivers’s {KEY_NAME}_file to read the password. However that require user’s to maintain the additional CSI inside of the same cluster.

The ideal solution will be the prometheus itself can resolve the secrets in itself and cache that for a configurable time, re-fetch the secrets after timeout
Similar alert manager issue: prometheus/alertmanager#3108

Proposed configuration

A new configuration type of <secret_config> will be introduced, the config will have the following fields so that users will use this to configure the type and id of the secrets to fetch from.

# "file", "aws_secret_manager", "azure_key_vault", etc.
type: <string>

aws_secret_manager: 
  
  # The arn of the secrets to fetch from
  secret_arn: <tmpl_string>
  
  # How long secrets stay in the memory. After timeout the secret will be refetch from the source
  [ timeout: <duration> | default = 5s ]
  
  # Configures AWS's Signature Verification 4 signing process to sign requests.
  sigv4:
    [ <sigv4_config> ]
  
file: 
  path: <filepath>

Example

aws_secret_manager type example

basic_auth:
  [ username: <string>]
  [ password_config: <secret_config>]

basic_auth:
  username: test
  password_config: 
    type: aws_secret_manager
    aws_secret_manager: 
      secret_arn: 'arn:aws:secretsmanager:us-west-2:111:secret:test-123'
      sigv4:
        region: us-west-2

Please let me know what do you think