Background

The Grafana Agent team is working on supporting the ServiceMonitor, PodMonitor, and Probe CRDs from Prometheus Operator directly in Grafana Agent.

This works by Grafana Agent discovering those resources in Kubernetes, and configuring an in-memory set of Prometheus components (discovery, scraper) to collect metrics from the corresponding targets.

We're able to map the CRDs to discovery and scraper configs for everything except the TLS configs, where the CA, cert, and key files must come from the local filesystem. As the various CRDs can reference Secrets for these fields, that would currently require Grafana Agent to read and save the contents of those secrets to the local filesystem of the container Grafana Agent is running on. I see this as a security risk.

Proposal

Ideally, we would be able to have an HTTPClientOption which allows us to pass a custom io/fs.FS instance for reading any files configured in the scrape configs. Example usage (from Grafana Agent) would be similar to the following:

scrape.NewManager(&scrape.Options{
  HTTPClientOptions: []config.HTTPClientOptions{
    config.WithFS(someFSInstance),
  },
}, ...)

This would allow us to retain the TLS certificates in-memory after reading them from the Kubernetes API and not have to sync them to the local filesystem.

This would enable using a virtual filesystem for the following configuration fields:

• basic_auth.password_file
• authorization.credentials_file
• oauth2.client_secret_file
• bearer_token_file
• tls_config.ca_file
• tls_config.cert_file
• tls_config.key_file

For the scope of this proposal, I'm suggesting that an HTTPClientOption is initially the only way to provide a custom io/fs.FS implementation; existing public API functions like config.NewTLSConfig would not be changed. The implementation of NewRoundTripperFromConfig will be updated to call new, unexported functions which do support a custom io/fs.FS implementation instead.

If an io/fs.FS instance is not provided, files would continue to be read using ioutil.ReadFile.

Would doing something like this be acceptable as a PR? If yes, I'll start working on it :)

cc @roidelapluie