feat: add secrets management by hsmatulis · Pull Request #17648 · prometheus/prometheus

hsmatulis · 2025-12-04T12:33:07Z

secrets: Add remote secrets providers

See the proposal here. This PR is currently a work in progress, demonstrating how user's would interact with the secrets management API

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

[FEATURE] config: All secret fields e.g. `scrape[].basic_auth.password` supports now "generic" format.  This means users are able to specify in `password` either inlined secret as previously or one from certain provider. Currently only `inline` and `file` providers are supported, with more to come. Old `password_field`-like fields are still supported although discouraged. This change also removes the undocumented `password_ref`-like fields which were unused. See [documentation](TODO) and [PROM-47](https://github.com/prometheus/proposals/blob/main/proposals/0047-secret_providers.md) for details.

bwplotka

This is good! Super minimal change surface.

I'd actually iterate over this PR and make this ready to be merged once common change is merged. We could double check all tests etc. 💪🏽

There are definitely some tests to fix, but it might be as trivial as adding (f *Field) Equals(other Field) bool so comparisons work correctly (they don't depend on state):

cmd/prometheus/main.go

bwplotka · 2025-12-08T14:40:58Z

discovery/kubernetes/kubernetes.go

 	NamespaceDiscovery NamespaceDiscovery      `yaml:"namespaces,omitempty"`
 	Selectors          []SelectorConfig        `yaml:"selectors,omitempty"`
 	AttachMetadata     AttachMetadataConfig    `yaml:"attach_metadata,omitempty"`
+	ExampleSecret      secrets.Field           `yaml:"example_secret"`


Nice, easy use 👍🏽

Let's remove and iterate over this PR to be effectively testable and potentially mergable once common change is done, looks promising!

Although your other examples use *secrets.Field (pointer)?

Switched to not using pointers as the default way, and added a nil provider to represent when the user passes in nothing

bwplotka · 2025-12-08T14:59:30Z

For release log, keep in mind this is for user facing changes (e.g. user of Prometheus YAML). I would mention what user can expect (generic field and removal of ref).

I'd recommend:

[FEATURE] config: All secret fields e.g. `scrape[].basic_auth.password` supports now "generic" format.  This means users are able to specify in `password` either inlined secret as previously or one from certain provider. Currently only `inline` and `file` providers are supported, with more to come. Old `password_field`-like fields are still supported although discouraged. This change also removes the undocumented `password_ref`-like fields which were unused. See [documentation](TODO) and [PROM-47](https://github.com/prometheus/proposals/blob/main/proposals/0047-secret_providers.md) for details.

bwplotka · 2025-12-11T08:04:21Z

Exciting feature! We even plan to mention it on KubeCon EU in March if that's ok (: cc @hsmatulis

Hopefully it's done until then 💪🏽

bwplotka · 2025-12-11T15:14:36Z

Relates to:

Copilot

Pull request overview

This PR adds secrets management functionality to Prometheus, allowing users to specify secrets from different providers (inline and file) instead of only inline secrets. The implementation includes a secrets manager component that runs as part of the main Prometheus process and populates configuration with resolved secrets during config loading and reloading.

Key Changes

Added secrets manager integration that runs alongside other Prometheus components
Modified the configuration reload mechanism to populate secrets from configured providers
Replaced prometheus/common dependency with a personal fork containing secrets management implementation

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File	Description
go.mod	Adds module replacement directing prometheus/common to personal fork with secrets management
go.sum	Updates dependency checksums for the replaced module and adds new indirect dependencies
cmd/prometheus/main.go	Integrates secrets manager lifecycle, adds context management, and modifies reload logic to populate secrets

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cmd/prometheus/main.go

Copilot · 2026-01-05T15:25:18Z

go.mod

 	google.golang.org/api v0.30.0
 )
+
+replace github.com/prometheus/common => github.com/hsmatulis/prom-common v0.0.8


The replace directive redirects the first-party module github.com/prometheus/common to a personal fork github.com/hsmatulis/prom-common v0.0.8, introducing a supply-chain risk because all imports of github.com/prometheus/common now execute code from an external, individually controlled repository. If that fork or its release process is compromised, an attacker could inject malicious logic into Prometheus, potentially exfiltrating secrets or altering security-critical behavior without changes in this repository. To reduce this risk, depend on the official github.com/prometheus/common module (or an internal fork under the Prometheus organization or vendored code) rather than a third-party fork as a drop-in replacement for core infrastructure code.

TODO: fix when prometheus/common#797 is merged

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

discovery/azure/azure.go

cmd/prometheus/main.go

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

discovery/ovhcloud/ovhcloud.go

discovery/ovhcloud/ovhcloud_test.go

Copilot

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Henrique Matulis <hmatulis@google.com>

bwplotka · 2026-01-15T16:18:56Z

discovery/aws/aws.go

 			c.EC2SDConfig.AccessKey = c.AccessKey
 		}
-		if c.SecretKey != "" {
+		if !c.SecretKey.IsNil() {


Suggested change

if !c.SecretKey.IsNil() {

if !c.SecretKey.IsEmpty() {

or

Suggested change

if !c.SecretKey.IsNil() {

if c.SecretKey.IsSet) {

bwplotka · 2026-01-15T16:38:21Z

discovery/openstack/instance_test.go

+		IdentityEndpoint:            s.Mock.Endpoint(),
+		ApplicationCredentialSecret: secrets.MockInline(""),


I'd vote to allow empty value and IsEmpty() on nil returning true and Value() returning empty string. Because we will likely forget to set it somewhere and then it's a bit annoying to have that extra verbose thing.

bwplotka reviewed Dec 8, 2025

View reviewed changes

bwplotka mentioned this pull request Dec 8, 2025

In progress proof of concept using secret library hsmatulis/prom-common#1

Draft

hsmatulis force-pushed the hsmatulis-secrets branch 18 times, most recently from 90d17c1 to 09ab0ec Compare January 5, 2026 13:20

hsmatulis marked this pull request as ready for review January 5, 2026 15:15

Copilot AI review requested due to automatic review settings January 5, 2026 15:15

Copilot started reviewing on behalf of hsmatulis January 5, 2026 15:15 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

hsmatulis force-pushed the hsmatulis-secrets branch 3 times, most recently from d77faac to 57e14f8 Compare January 14, 2026 08:34

hsmatulis requested a review from Copilot January 14, 2026 08:35

Copilot started reviewing on behalf of hsmatulis January 14, 2026 08:35 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

discovery/azure/azure.go Outdated Show resolved Hide resolved

cmd/prometheus/main.go Outdated Show resolved Hide resolved

hsmatulis force-pushed the hsmatulis-secrets branch 2 times, most recently from e5e361e to 72cf435 Compare January 14, 2026 10:33

hsmatulis requested a review from Copilot January 14, 2026 10:34

Copilot AI reviewed Jan 14, 2026

View reviewed changes

discovery/ovhcloud/ovhcloud.go Show resolved Hide resolved

discovery/ovhcloud/ovhcloud_test.go Outdated Show resolved Hide resolved

hsmatulis force-pushed the hsmatulis-secrets branch 2 times, most recently from 5bf50e3 to f2279c4 Compare January 14, 2026 10:56

hsmatulis requested a review from Copilot January 14, 2026 10:57

Copilot AI reviewed Jan 14, 2026

View reviewed changes

feat: add secrets management

a95868d

Signed-off-by: Henrique Matulis <hmatulis@google.com>

hsmatulis force-pushed the hsmatulis-secrets branch from f2279c4 to a95868d Compare January 14, 2026 11:07

bwplotka reviewed Jan 15, 2026

View reviewed changes

austindyoung mentioned this pull request Feb 26, 2026

[mirror] prometheus/prometheus#17648 feat: add secrets management austindyoung/prometheus-driftfence-mirror-fork#157

Open

		IdentityEndpoint: s.Mock.Endpoint(),
		ApplicationCredentialSecret: secrets.MockInline(""),

Conversation

hsmatulis commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

secrets: Add remote secrets providers

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

bwplotka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwplotka Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

bwplotka Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

hsmatulis Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

bwplotka commented Dec 8, 2025

Uh oh!

bwplotka commented Dec 11, 2025

Uh oh!

bwplotka commented Dec 11, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

hsmatulis Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

bwplotka Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

bwplotka Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hsmatulis commented Dec 4, 2025 •

edited

Loading

bwplotka left a comment •

edited

Loading