Skip to content

feat: introduce secret rewrite merge operation#4894

Merged
Skarlso merged 17 commits intoexternal-secrets:mainfrom
riccardomc:2987-rewrite-merge
Jun 24, 2025
Merged

feat: introduce secret rewrite merge operation#4894
Skarlso merged 17 commits intoexternal-secrets:mainfrom
riccardomc:2987-rewrite-merge

Conversation

@riccardomc
Copy link
Copy Markdown
Member

@riccardomc riccardomc commented Jun 11, 2025

Problem Statement

This pull request implements the suggestion in #2987 which allows to merge secret values from different secrets obtained from a provider through a dataFrom.find in one single Secret according to configurable priority, merge strategy and conflict policy.

Related Issue

Implements #2987

Proposed Changes

The implementation extends the ExternalSecretDataFromRemoteRef API by introducing a new ExternalSecretRewriteMerge method to instruct the controller on how to perform merge operations on secret key/value pairs as part of ExternalSecretRewrite.

The implementation follows this detailed suggestion comment for the most part, with some interpretations and choices.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  # ...
  dataFrom:
  - find:
      path: cluster1/backend
      name: 
        regexp: ".*"
    rewrite:
    - merge:
        # define the key under which we want to store the resulting merged values
        # It is required if the strategy is 'JSON' and ignored if strategy is 'Extract' 
        into: ""

        # JSON: keep the keys (secret names from the provider)
        # and only merge the secret values, treating it as JSON.
        # The resulting JSON object is stored in the `into` property.
        #
        # Extract: parse the value as JSON and hoist
        # the key/value pairs up just like dataFrom.Extract it does.
        strategy: Extract # or JSON

        # Control the behavior when a conflicting key/value is encountered
        # When conflictPolicy=Ignore the conflicting key/value will be 
        # silently overridden. When conflictPolicy=Error the conflict will cause
        # an Error that will be reported in the Status of the ExternalSecret with
        # details about all conflicting keys.
        conflictPolicy: Error # or Ignore

        # define a list of keys that take precedence over all keys,
        # no matter what sorting strategy/order is being used.
        # These priority-keys are merged into the result map
        # in the defined order.
        priority:
          - custom-stuff-controlled-by-developers

Found secret keys are sorted in alphabetical order before being processed. This guarantees a stable and predictable conflict resolution order. Keys specified in the priority list are processed after all the others, guaranteeing that the last key on the priority list has precedence over all the others.

Documentation and extensive unit tests are provided to further illustrate the implementation details.

Additional Notes

  • This is my first contribution in a long while, so please be patient with me, ok? 😘
  • There are a couple of failing tests that might depend on my local environment since I get the same failures on main. So I would like to get the ball rolling nonetheless and see if they pass in CI (update: passed...). Here's a snippet of the locally failing tests:
 [FAILED] in [It] - /Users/rmc/dev/external-secrets/pkg/controllers/secretstore/common_test.go:69 @ 06/11/25 21:19:48.454
  << Timeline

  [FAILED] Timed out after 10.001s.
  Expected
      <bool>: false
  to be true
  In [It] at: /Users/rmc/dev/external-secrets/pkg/controllers/secretstore/common_test.go:69 @ 06/11/25 21:19:48.454
------------------------------
•••

Summarizing 2 Failures:
  [FAIL] SecretStore reconcile Controller Reconcile logic [It] [namespace] invalid provider with secretStore should set InvalidStore condition
  /Users/rmc/dev/external-secrets/pkg/controllers/secretstore/common_test.go:69
  [FAIL] SecretStore reconcile Controller Reconcile logic [It] [cluster] invalid provider with secretStore should set InvalidStore condition
  /Users/rmc/dev/external-secrets/pkg/controllers/secretstore/common_test.go:69

Ran 8 of 8 Specs in 40.041 seconds
FAIL! -- 6 Passed | 2 Failed | 0 Pending | 0 Skipped

Checklist

  • I have read the contribution guidelines
  • All commits are signed with git commit --signoff
  • My changes have reasonable test coverage
  • All tests pass with make test
  • I ensured my PR is ready for review with make reviewable

@riccardomc riccardomc requested a review from a team as a code owner June 11, 2025 19:31
@riccardomc riccardomc requested a review from Skarlso June 11, 2025 19:31
@gusfcarvalho
Copy link
Copy Markdown
Member

why the hell did helm-release run here? 🤔

@riccardomc
Copy link
Copy Markdown
Member Author

riccardomc commented Jun 12, 2025

why the hell did helm-release run here? 🤔

I didn't do nothing, I swear! 🥲

A few quesitons:

  • Do you want me to refactor this issue by sonarqube? I'd be happy to if necessary
  • Any idea why tests I pointed out in the description fail locally?

@riccardomc riccardomc force-pushed the 2987-rewrite-merge branch from a4614b8 to ef98618 Compare June 13, 2025 09:06
Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
…writes

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
…rations

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
@riccardomc riccardomc force-pushed the 2987-rewrite-merge branch from ef98618 to d6979f3 Compare June 13, 2025 09:07
… merge

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
@riccardomc riccardomc force-pushed the 2987-rewrite-merge branch from 15c387d to fbdc2b1 Compare June 13, 2025 09:56
@Skarlso
Copy link
Copy Markdown
Contributor

Skarlso commented Jun 13, 2025

@riccardomc Yes please, if you could take care of the sonar issue that would be fantastic. :) 🙇

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
@riccardomc
Copy link
Copy Markdown
Member Author

@Skarlso sure, here you go! 😉

@Skarlso
Copy link
Copy Markdown
Contributor

Skarlso commented Jun 14, 2025

Thanks! :)

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
@riccardomc riccardomc force-pushed the 2987-rewrite-merge branch from a127ba2 to 4945ea8 Compare June 15, 2025 20:03
@sonarqubecloud
Copy link
Copy Markdown

@Skarlso Skarlso merged commit d6d5ce9 into external-secrets:main Jun 24, 2025
23 checks passed
@Aransh
Copy link
Copy Markdown
Contributor

Aransh commented Jun 26, 2025

This looks incredible!
Will save me so much work and current headaches, thanks a lot for your work!

@Skarlso
Copy link
Copy Markdown
Contributor

Skarlso commented Jun 27, 2025

Nice. :) This has been released with 0.18.1. :) Enjoy!

alliseeisgold pushed a commit to alliseeisgold/external-secrets that referenced this pull request Jul 10, 2025
* initial implementation of rewrite merge

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* introduce stable secrets merging order to ensure predictable key overwrites

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* dataFrom.rewrite.merge: implement key priority

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* introduce conflictPolicy & strategy, refactor

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* polish code, improve tests, add combined rewrite tests

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* add documentation

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* make Error default conflict policy for ExternalSecretRewriteMerge operations

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* update CRDs after setting Error as default conflictPolicy for rewrite merge

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* reduce cognitive complexity of RewriteMap method

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* improve error messages by adding failed rewrite operation type

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* use maps.Copy() in RewriteMerge to propagate input map

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

* introduce check if key in priority list exists in input

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>

---------

Signed-off-by: Riccardo M. Cefala <riccardo.c@miro.com>
Co-authored-by: Gergely Brautigam <182850+Skarlso@users.noreply.github.com>
Signed-off-by: asrormirzoev <asrormirzoev@yandex-team.ru>
@Aransh
Copy link
Copy Markdown
Contributor

Aransh commented Sep 15, 2025

@riccardomc Just wanted to say I finally had the chance to start working with this today and it's working perfectly!
Thanks again for your contribution 🙏

@Aransh
Copy link
Copy Markdown
Contributor

Aransh commented Sep 15, 2025

Quick update
While looking into migrating to this approach I stumbled into a bit of a soft blocker.
To quote #2987 (comment) - "it would be great to be able to specify priority for specific secret(s), if they exist"

Seems like this implementation is missing the "if they exist" part, as if I try to create an externalSecret with a "priority" configuration, but the key specified does not exist, it will result in an error.

For example, for this:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-secrets
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: hashicorp-vault-backend
  target:
    name: my-app-secrets
  dataFrom:
  - find:
      path: my-app
      name:
        regexp: ".*"
    rewrite:
    - merge:
        strategy: Extract
        conflictPolicy: Ignore
        priority:
          - my-app/secrets

If "my-app/secrets" does not exist, will result in:

Warning UpdateFailed 26s (x2 over 48s) external-secrets error processing spec.dataFrom[0].find, err: error applying rewrite to keys: failed rewrite operation[0]: merge failed with key "my-app/secrets" not found in input map

While I CAN see value in this causing an error if a secret is missing in some use-cases (for example if there's a secret you want to mandate), I think the "this key is prioritized if it exists" approach makes more sense, at least as the default.
Other solution for me would be requiring all apps create a blank "prioritized" secret (and I'm talking hundreds of apps 😵‍💫), so I think I'll hold off with this migration for now unfortunately

@riccardomc May I please request a follow up to either make prioritized secrets optional, or if you see fit, add a "priorityStrategy" field for either "Error" or "Ignore" 👉👈

@riccardomc
Copy link
Copy Markdown
Member Author

Hey, thanks! I am very happy you're using it and thanks for reaching out, much appreciated.

The main reason I decided to keep this behavior is to minimize surprises for end users: if we just ignore the non existing remote secret, then the final content in the Secret might be different than expected without very little possibility of noticing.

I thought that this could lead to very unpleasant circumstances: it's quite easy to fail to override credentials that might cause significant issues at the application level. Imagine failing to override Production connection string because the Staging remote secret has a typo.

Maybe the priorityStrategy suggestion is the right thing to do, so at least there's a conscious opt-in into this behavior.

Would you mind opening a separate Issue for this as this is now merged?

@Aransh
Copy link
Copy Markdown
Contributor

Aransh commented Sep 16, 2025

@riccardomc sure, will do, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants