Raise error when unknown key specified in template by shuheiktgw · Pull Request #3480 · external-secrets/external-secrets

shuheiktgw · 2024-05-12T01:51:22Z

Problem Statement

While working on the template, I noticed if I specify a non-existent key in the template, the reconciliation succeeds and inserts value <no value>. This behavior seems quite error-prone since users could accidentally insert base64 encoded <no value> to their secret value.

Related Issue

N/A

Proposed Changes

Use the missingkey=error option for the go template to raise the error.

Checklist

I have read the contribution guidelines
All commits are signed with git commit --signoff
My changes have reasonable test coverage
All tests pass with make test
I ensured my PR is ready for review with make reviewable

Signed-off-by: shuheiktgw <s-kitagawa@mercari.com>

gusfcarvalho · 2024-05-13T11:31:38Z

IIRC this was set on purpose - @moolen do you happen to know what was the reason we did that on purpose?

moolen · 2024-05-13T18:43:07Z

I don't know, it doesn't ring a bell. From the looks of it i think this is fine.

gusfcarvalho · 2024-05-13T21:12:34Z

In that case, I think we need to update documentation as well in https://external-secrets.io/latest/guides/templating/:

base64encode was renamed to b64enc.
base64decode was renamed to b64dec. Any errors that occur during decoding are silenced.
fromJSON was renamed to fromJson. Any errors that occur during unmarshalling are silenced.
toJSON was renamed to toJson. Any errors that occur during marshalling are silenced.

As these are no longer silenced if I understand correctly 😄

shuheiktgw · 2024-05-13T23:38:14Z

Thanks for the feedback! IIUC, the doc states errors returned from those three functions are ignored, I personally believe we don't have to change that part since we keep ignoring those errors even after the change, but WDYT? 🤔

gusfcarvalho · 2024-05-14T15:46:28Z

wouldn't a (fromJson .).key start failing now?

shuheiktgw · 2024-05-15T00:40:33Z

wouldn't a (fromJson .).key start failing now?

Yes, that is correct! At the same time, the current statement "fromJSON was renamed to fromJson. Any errors that occur during unmarshalling are silenced." also stays true even after this change. Why don't we simply update the doc and tell users that any missing keys raise an error?

gusfcarvalho · 2024-05-15T07:50:30Z

wouldn't a (fromJson .).key start failing now?

Yes, that is correct! At the same time, the current statement "fromJSON was renamed to fromJson. Any errors that occur during unmarshalling are silenced." also stays true even after this change. Why don't we simply update the doc and tell users that any missing keys raise an error?

works for me. :) sorry for nitting

…g keys Signed-off-by: shuheiktgw <s-kitagawa@mercari.com>

sonarqubecloud · 2024-05-16T23:17:01Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.5% Duplication on New Code

See analysis details on SonarCloud

shuheiktgw · 2024-05-16T23:18:26Z

@gusfcarvalho Thanks, updated the docs!

gusfcarvalho · 2024-05-17T12:57:09Z

I cannot be more grateful by having you as maintainer 😄

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [external-secrets](https://github.com/external-secrets/external-secrets) | patch | `0.9.18` -> `0.9.19` | --- ### Release Notes <details> <summary>external-secrets/external-secrets (external-secrets)</summary> ### [`v0.9.19`](https://github.com/external-secrets/external-secrets/releases/tag/v0.9.19) [Compare Source](external-secrets/external-secrets@v0.9.18...v0.9.19) Image: `ghcr.io/external-secrets/external-secrets:v0.9.19` Image: `ghcr.io/external-secrets/external-secrets:v0.9.19-ubi` Image: `ghcr.io/external-secrets/external-secrets:v0.9.19-ubi-boringssl` #### What's Changed - bump 0.9.18 by [@gusfcarvalho](https://github.com/gusfcarvalho) in external-secrets/external-secrets#3492 - Fix flaky ES controller test by [@shuheiktgw](https://github.com/shuheiktgw) in external-secrets/external-secrets#3493 - Raise error when unknown key specified in template by [@shuheiktgw](https://github.com/shuheiktgw) in external-secrets/external-secrets#3480 - Update ci.yml with ppc64le arch support by [@IdanAdar](https://github.com/IdanAdar) in external-secrets/external-secrets#3500 - chore(deps): bump regex from 2024.5.10 to 2024.5.15 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3512 - chore(deps): bump zipp from 3.18.1 to 3.18.2 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3511 - chore(deps): bump platformdirs from 4.2.1 to 4.2.2 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3510 - chore(deps): bump golang from `6d71b7c` to `5c56bd4` in /e2e by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3508 - chore(deps): bump github/codeql-action from 3.25.4 to 3.25.5 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3507 - chore(deps): bump codecov/codecov-action from 4.3.1 to 4.4.0 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3506 - chore(deps): bump actions/checkout from 4.1.5 to 4.1.6 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3505 - chore(deps): bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3504 - chore(deps): bump golang from `2a88224` to `f1fe698` by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3503 - chore(deps): bump mkdocs-material from 9.5.22 to 9.5.23 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3509 - chore: update dependencies by [@eso-service-account-app](https://github.com/eso-service-account-app) in external-secrets/external-secrets#3513 - ppc64le additions by [@IdanAdar](https://github.com/IdanAdar) in external-secrets/external-secrets#3535 - chore(deps): bump ubi8/ubi-minimal from `2fa47fa` to `f729a7f` by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3523 - doc(BitWarden): extends the liveness timeout by [@acolombier](https://github.com/acolombier) in external-secrets/external-secrets#3542 - chore(deps): bump mkdocs-material from 9.5.23 to 9.5.24 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3526 - chore(deps): bump zipp from 3.18.2 to 3.19.0 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3527 - chore(deps): bump requests from 2.31.0 to 2.32.2 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3528 - chore(deps): bump codecov/codecov-action from 4.4.0 to 4.4.1 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3530 - chore(deps): bump aquasecurity/trivy-action from 0.20.0 to 0.21.0 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3531 - chore(deps): bump github/codeql-action from 3.25.5 to 3.25.6 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3532 - chore(deps): bump ubi8/ubi-minimal from `f729a7f` to `9e458f4` by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3543 - chore(deps): bump golang from `f1fe698` to `b8ded51` by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3522 - chore(deps): bump alpine from 3.19.1 to 3.20.0 in /e2e by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3525 - chore(deps): bump watchdog from 4.0.0 to 4.0.1 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3529 - chore(deps): bump alpine from 3.19 to 3.20 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3533 - chore: update dependencies by [@eso-service-account-app](https://github.com/eso-service-account-app) in external-secrets/external-secrets#3534 - Fix: ESO template crash when Kubernetes secret data is nil by [@ma-ble](https://github.com/ma-ble) in external-secrets/external-secrets#3537 - chore(deps): bump alpine from `c5b1261` to `77726ef` by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3524 - Add CA-Bundle to pemToPkcs12 output by [@WatcherWhale](https://github.com/WatcherWhale) in external-secrets/external-secrets#3494 - chore(deps): bump docker/login-action from 3.1.0 to 3.2.0 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3544 - chore(deps): bump github/codeql-action from 3.25.6 to 3.25.7 by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3545 - chore(deps): bump certifi from 2024.2.2 to 2024.6.2 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3546 - chore(deps): bump mkdocs-material from 9.5.24 to 9.5.25 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3547 - chore(deps): bump zipp from 3.19.0 to 3.19.1 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3549 - chore(deps): bump requests from 2.32.2 to 2.32.3 in /hack/api-docs by [@dependabot](https://github.com/dependabot) in external-secrets/external-secrets#3548 - chore: update dependencies by [@eso-service-account-app](https://github.com/eso-service-account-app) in external-secrets/external-secrets#3550 #### New Contributors - [@acolombier](https://github.com/acolombier) made their first contribution in external-secrets/external-secrets#3542 - [@ma-ble](https://github.com/ma-ble) made their first contribution in external-secrets/external-secrets#3537 **Full Changelog**: external-secrets/external-secrets@v0.9.18...v0.9.19 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.internal/nrdufour/home-ops/pulls/471 Co-authored-by: Renovate <renovate@ptinem.casa> Co-committed-by: Renovate <renovate@ptinem.casa>

qlonik · 2024-06-06T02:19:26Z

Hey, just wanted to ask about this. With this change it is now not possible to use .missing_key | default "some-default" anymore, since it throws an error when accessing the missing key. Is there any way to make | default work?

Because of the changes in external-secrets/external-secrets#3480, external-secrets controller fails when template tries to use a key that does not exist (even though there is a `| default` filter in-use).

shuheiktgw · 2024-06-06T02:40:37Z

@qlonik Hmm, yes that is correct... This is more like a Go template level problem, so there must be some ways to workaround the problem so let me check 🙇

shuheiktgw · 2024-06-06T21:40:47Z

@qlonik I've figured out how we can set the default values without raising errors. If this is a common use case, it might be favorable to add syntax sugar for it...

{{ if index . "missing_key" }}{{ .missing_key }}{{ else }}some-default{{ end }}

…3480) * Raise error when unknown key specified in template Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> * Update the template docs to clarify the new behavior with non-existing keys Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> --------- Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> Signed-off-by: Bude8 <henryblee8@gmail.com>

qlonik · 2024-06-21T03:14:26Z

Hey @shuheiktgw, sorry, I haven't responded sooner. I don't know if this is necessary - I only had a few spots where this was an issue, and I just inlined default values where it was necessary.

However, I also found dig function, which I think can be used here and also can be used in the pipe:

{{ dig "missing_key" "some-default" . }}
{{ . | dig "missing_key" "some-default" }}
{{ . | ... | dig "missing_key" "some-default" | ... }}

…3480) * Raise error when unknown key specified in template Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> * Update the template docs to clarify the new behavior with non-existing keys Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> --------- Signed-off-by: shuheiktgw <s-kitagawa@mercari.com> Signed-off-by: Bill Hamilton <bill.hamilton@delinea.com>

ghost · 2024-07-08T22:44:38Z

Hi.

I think this broke my push secret as prior to upgrading this was working for me. I create a secret from a k8s secret on top of data I set using the templating and I get the error of:

external-secrets {"level":"error","ts":1720450212.1143377,"msg":"Reconciler error","controller":"pushsecret","controllerGroup":"[external-secrets.io](http://external-secrets.io/)","controllerKind":"PushSecret","PushSecret":{"name":"rabbitmq","namespace":"rabbitmq"},"namespace":"rabbitmq","name":"rabbitmq","reconcileID":"<REDACTED>","error":"could not execute template: could not execute template: unable to execute template at key endpoint: unable to execute template at key endpoint: template: endpoint:1:3: executing \"endpoint\" at <.endpoint>: map has no entry for key \"endpoint\"","stacktrace":"[sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler)\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:324\[nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem](http://nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem)\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:261\[nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2](http://nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2)\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:222"}

This is my spec

Spec:
  Data:
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         endpoint
        Remote Key:       rabbitmq
      Secret Key:         host
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         endpoint-scheme
        Remote Key:       rabbitmq
      Secret Key:         endpoint-scheme
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         management-port
        Remote Key:       rabbitmq
      Secret Key:         management-port
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         management-scheme
        Remote Key:       rabbitmq
      Secret Key:         management-scheme
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         port
        Remote Key:       rabbitmq
      Secret Key:         port
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:         password
        Remote Key:       rabbitmq
      Secret Key:         password
    Conversion Strategy:  None
    Match:
      Remote Ref:
        Property:    username
        Remote Key:  rabbitmq
      Secret Key:    username
  Deletion Policy:   None
  Refresh Interval:  1h
  Secret Store Refs:
    Kind:  ClusterSecretStore
    Name:  cluster-secret-store
  Selector:
    Secret:
      Name:  rabbitmq-default-user
  Template:
    Data:
      Endpoint:             {{ .endpoint }}
      Endpoint - Scheme:    amqp
      Management - Port:    15672
      Management - Scheme:  http
      Password:             {{ .password }}
      Port:                 {{ .port }}
      Username:             {{ .username }}
    Engine Version:         v2
    Merge Policy:           Replace
  Update Policy:            Replace

The secret should be

{
  "endpoint": "rabbitmq.rabbitmq.svc",
  "endpoint-scheme": "amqp",
  "management-port": "15672",
  "management-scheme": "http",
  "port": "5672",
  "password": "<REDACTED>",
  "username": "<REDACTED>"
}

What's weird is if I set

Endpoint:             {{ .endpoint }}

to

Endpoint:             {{ .host }}

It now works.... ?

Raise error when unknown key specified in template

b9dc1fe

Signed-off-by: shuheiktgw <s-kitagawa@mercari.com>

shuheiktgw requested a review from a team as a code owner May 12, 2024 01:51

shuheiktgw requested a review from rogertuma May 12, 2024 01:51

Skarlso previously approved these changes May 16, 2024

View reviewed changes

Update the template docs to clarify the new behavior with non-existin…

27206c9

…g keys Signed-off-by: shuheiktgw <s-kitagawa@mercari.com>

shuheiktgw dismissed Skarlso’s stale review via 27206c9 May 16, 2024 23:15

gusfcarvalho approved these changes May 17, 2024

View reviewed changes

gusfcarvalho merged commit b156e23 into external-secrets:main May 17, 2024

shuheiktgw deleted the fail_for_unknown_key branch June 6, 2024 02:35

Skarlso mentioned this pull request Sep 23, 2024

Regression in v0.9.19 when using 'default' in secret templates #3843

Closed

Uh oh!

Conversation

shuheiktgw commented May 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Related Issue

Proposed Changes

Checklist

Uh oh!

gusfcarvalho commented May 13, 2024

Uh oh!

moolen commented May 13, 2024

Uh oh!

gusfcarvalho commented May 13, 2024

Uh oh!

shuheiktgw commented May 13, 2024

Uh oh!

gusfcarvalho commented May 14, 2024

Uh oh!

shuheiktgw commented May 15, 2024

Uh oh!

gusfcarvalho commented May 15, 2024

Uh oh!

sonarqubecloud bot commented May 16, 2024

Quality Gate passed

Uh oh!

shuheiktgw commented May 16, 2024

Uh oh!

gusfcarvalho commented May 17, 2024

Uh oh!

qlonik commented Jun 6, 2024

Uh oh!

shuheiktgw commented Jun 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shuheiktgw commented Jun 6, 2024

Uh oh!

qlonik commented Jun 21, 2024

Uh oh!

ghost commented Jul 8, 2024 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shuheiktgw commented May 12, 2024 •

edited

Loading

shuheiktgw commented Jun 6, 2024 •

edited

Loading

ghost commented Jul 8, 2024 •

edited by ghost

Loading