Add enhanced sensitive data handling by kuzaxak · Pull Request #364 · werf/nelm

kuzaxak · 2025-06-14T17:42:41Z

Functionality adds similar to helm-diff style of diffing sensitive
data.

Previously, NELM handled sensitive data with an all-or-nothing approach:

Resources marked as sensitive (via werf.io/sensitive: "true") were
completely hidden
Secrets were entirely hidden, providing no visibility into structural changes
No way to selectively redact only sensitive fields while showing the rest

This made it difficult to:

Track changes in Secret keys or metadata
See the size of sensitive data changes
Selectively protect only truly sensitive fields in complex resources

Added support for werf.io/sensitive-paths annotation that accepts a
JSONPath expressions:

metadata:
 annotations:
  werf.io/sensitive-paths: $.spec.template.spec.containers[*].env[?(@.name=='API_KEY')].value,$.data.password

Sensitive values are replaced with size information:

password: SENSITIVE (len 12 bytes)

I used ojg library as JSONPath as kube client do not support
native mutation for complex structures like slices.

kuzaxak · 2025-06-14T17:48:00Z

@ilya-lesikov Hey, I hope it's alright to propose a change without creating an issue. We tried to use Nelm, but the existing sensite diff functionality is rather limited.

ilya-lesikov · 2025-06-16T09:15:39Z

@kuzaxak Hi, thanks for the PR, looks interesting. I'll look into it. Would've been better to create an issue first so we decide on the design, but whatever.

kuzaxak · 2025-06-16T10:28:30Z

@kuzaxak Hi, thanks for the PR, looks interesting. I'll look into it. Would've been better to create an issue first so we decide on the design, but whatever.

Thanks! I was thinking that it might be better to reuse existing annotation and determine, based on the type, if it's a bool, then use the hide all function; if it's a string that can be parsed into an array, then treat it as a JSON path.

ilya-lesikov · 2025-06-25T11:59:54Z

@kuzaxak

Hi, looked through it, love the feature, thank you. Can we make a few design changes?

I think this feature is so flexible that it makes "werf.io/sensitive" kinda obsolete. How about we do this:

Add new featgate NELM_FEAT_FIELD_SENSITIVE=true via featgate package.
NELM_FEAT_FIELD_SENSITIVE=true will be the default in v2, so let's activate NELM_FEAT_FIELD_SENSITIVE by enabling NELM_FEAT_PREVIEW_V2.
Can we change the format a bit? werf.io/sensitive-paths: "data.password,spec.template.spec.containers.*.env.*.value", in line with all the other annotations. \, should escape the comma, so we don't split entries on it.
If NELM_FEAT_FIELD_SENSITIVE=true and the resource has werf.io/sensitive: "true" annotation, then it works as werf.io/sensitive-paths: "spec.*,data.*", otherwise current behavior
If NELM_FEAT_FIELD_SENSITIVE=true and the resource kind is Secret and it has no werf.io/sensitive or werf.io/sensitive-paths annotations, then it works as werf.io/sensitive-paths: "data.*", otherwise current behavior
Let's use the actual JSONPath parser, for example "k8s.io/client-go/util/jsonpath", which we already use in werf/kubedog. Grok returned me this PoC, no idea if this works, but the general idea is right: https://pastebin.com/1bSFqza2 If "k8s.io/client-go/util/jsonpath" doesn't fit we can find something else.
Sometimes we replace "REDACTED" with "REDACTED" or "REDACTED (len 5 bytes)" with "REDACTED (len 5 bytes)", both of which produce no diff, which results in <hidden insignificant changes>. How about we compute SHA256 hash and truncate it to 12 chars, then show this: SENSITIVE (5 bytes, e0c9035898dd). We also need to do this for slices and maps: do len(slice/map), then marshal the slice/map into the JSON string and get its SHA256, then show this: SENSITIVE (10 entries, e0c9035898dd).
Also make sure that when type of a value changes (e. g. string -> slice) or when value is deleted/added as a whole, then the correct diffs are produced.

That's quite a few changes, but if this is done I'll merge it. I'll probably need to add CONTRIBUTORS.md and mention that it's best if we decide on the design first, since it's not easy to come up with it from the get-go. Pretty close overall, thanks again.

kuzaxak · 2025-06-25T15:04:58Z

Hi, looked through it, love the feature, thank you. Can we make a few design changes?

Ofc, this is just my initial proposal.

Can we change the format a bit? werf.io/sensitive-paths: "data.password,spec.template.spec.containers..env..value"
Let's use the actual JSONPath parser, for example "k8s.io/client-go/util/jsonpath",

To clarify, JSON path or glob string like spec.template.spec.containers.*.env.*.value? JSON path would be like $.spec.template.spec.containers[*].env[*].value.

How about we compute SHA256 hash and truncate it to 12 chars, then show this: SENSITIVE (5 bytes, e0c9035898dd)

Nice idea! I will do all the listed changes.

I'll probably need to add CONTRIBUTORS.md and mention that it's best if we decide on the design first, since it's not easy to come up with it from the get-go.

Do you expect a discussion to be opened or just an issue? As I have a few more proposals :)

@ilya-lesikov

ilya-lesikov · 2025-06-25T18:08:52Z

@kuzaxak

To clarify, JSON path or glob string like spec.template.spec.containers.*.env.*.value? JSON path would be like $.spec.template.spec.containers[*].env[*].value.

Well, JSONPath does look kinda uglier. But at least it's flexible, unambiguous and standard. Let's stick to the JSONPath version. We'll compensate for it in the docs. And we'll stick to JSON Path in all the other features like this one, when the user needs to select some fields in JSON/YAML.

Nice idea! I will do all the listed changes.

Great :)

Do you expect a discussion to be opened or just an issue? As I have a few more proposals :)

An issue is fine.

kuzaxak · 2025-06-25T18:56:39Z

@ilya-lesikov Done. I implemented full JSONPath support separated by commas and added some tests to ensure various structures are replaced correctly. Unfortunately, kube API JSONPath isn't working well with slices, and I used an external library that handles that.

Hashing works too, now diff detects the value change even if the length matches.

kuzaxak · 2025-06-30T16:44:20Z

@ilya-lesikov hey, could you please check it?

README.md

internal/resource/sensitive.go

go.mod

ilya-lesikov · 2025-07-04T19:43:20Z

@kuzaxak Sorry, was pretty busy lately. Looks good, only few changes needed.

Functionality adds similar to [helm-diff][1] style of diffing sensitive data. Previously, NELM handled sensitive data with an all-or-nothing approach: - Resources marked as sensitive (via `werf.io/sensitive: "true"`) were completely hidden - Secrets were entirely hidden, providing no visibility into structural changes - No way to selectively redact only sensitive fields while showing the rest This made it difficult to: - Track changes in Secret keys or metadata - See the size of sensitive data changes - Selectively protect only truly sensitive fields in complex resources Added support for `werf.io/sensitive-paths` annotation that accepts a JSONPath expressions: ```yaml metadata: annotations: werf.io/sensitive-paths: $.spec.template.spec.containers[*].env[?(@.name=='API_KEY')].value,$.data.password ``` Sensitive values are replaced with size information: ``` password: SENSITIVE (len 12 bytes) ``` I used [ojg][2] library as JSONPath as kube client do not support native mutation for complex structures like slices. [1]: https://github.com/databus23/helm-diff [2]: https://github.com/ohler55/ojg Signed-off-by: Vladimir Kuznichenkov <kuzaxak.tech@gmail.com>

kuzaxak · 2025-07-10T17:47:23Z

@ilya-lesikov implemented the last comments. Could you please check one more time?

ilya-lesikov

Tried it out, the only issue I found is this:

❯ cat chart/templates/test.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: to-create
  annotations:
    werf.io/sensitive-paths: '$.data[*]'
data:
  key1: {{ uuidv4 }}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: to-update
  annotations:
    werf.io/sensitive-paths: '$.data[*]'
data:
  key1: {{ uuidv4 }}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: to-delete
  annotations:
    werf.io/sensitive-paths: '$.data[*]'
data:
  key1: {{ uuidv4 }}

❯ nelm release install -n jkldfjkdkjjkdk -r jkldfj .helm

❯ NELM_FEAT_FIELD_SENSITIVE=false nelm release plan install -n jkldfjkdkjjkdk -r jkldfj .helm
Planning release install "jkldfj" (namespace: "jkldfjkdkjjkdk")

┌ Create ConfigMap/to-create
│ + apiVersion: v1
│ + kind: ConfigMap
│ + metadata:
│ +   name: to-create
│ +   namespace: ""
└ Create ConfigMap/to-create

┌ Update ConfigMap/to-delete
│ <hidden insignificant changes>
└ Update ConfigMap/to-delete

┌ Update ConfigMap/to-update
│ <hidden insignificant changes>
└ Update ConfigMap/to-update

Planned changes summary for release "jkldfj" (namespace: "jkldfjkdkjjkdk"):
- create: 1 resource(s)
- update: 2 resource(s)

Otherwise works great. If you push the fix, please don't force-push it, just add new commits, this way it's easier for me to review only the new changes. I'll squash all the extra commits just before merging.

kuzaxak · 2025-07-11T12:34:30Z

To clarify, you mean that the resource got hidden as

┌ Update ConfigMap/to-update
│ <hidden insignificant changes>
└ Update ConfigMap/to-update

And in the case of creation, it renders metadata only without a data section?

kuzaxak · 2025-07-11T12:36:12Z

By the way, do you have any plans to create integration tests? I see you have some frameworks in dependencies. Perhaps it is worth covering this functionality with tests.

I've added a small set of unit tests, but I am not sure if it is adequate and matches your expectations.

ilya-lesikov · 2025-07-11T12:47:33Z

Tests are good, don't bother. Ultimately I was planning to use ginkgo/gomega for tests, like we do in werf and other places, e.g.: https://github.com/deckhouse/delivery-kit-sdk/blob/main/pkg/signature/elf/inhouse/file_test.go
https://github.com/werf/werf/blob/main/test/e2e/converge/simple_test.go

But that's not a priority now. Nelm is mostly tested in werf for now, since nelm is the werf deployment engine, so if anything breaks I usually see it there.

Feature flag shouldn't impact the way how we are redacting objects using a new annotation.

kuzaxak · 2025-07-11T13:41:08Z

@ilya-lesikov fixed a bug and covered it with a test to validate. Missed that case originally, thanks for checking!

ilya-lesikov · 2025-07-11T13:57:16Z

@kuzaxak Merged, thanks!

…E featgate (#364) Just to trigger release, the actual commit is 7860bc2 Signed-off-by: Ilya Lesikov <ilya@lesikov.com>

kuzaxak force-pushed the configurable-diff branch from 5c3ca4b to ece6f8c Compare June 14, 2025 17:43

kuzaxak force-pushed the configurable-diff branch 2 times, most recently from 730ae12 to 7f99f12 Compare June 25, 2025 18:54

ilya-lesikov requested changes Jul 4, 2025

View reviewed changes

kuzaxak force-pushed the configurable-diff branch from 7f99f12 to 1100c92 Compare July 10, 2025 17:43

kuzaxak force-pushed the configurable-diff branch from 1100c92 to e55ff90 Compare July 10, 2025 17:45

kuzaxak requested a review from ilya-lesikov July 10, 2025 17:47

ilya-lesikov requested changes Jul 11, 2025

View reviewed changes

Remove feature gate check from sensitive data redaction

66740c5

Feature flag shouldn't impact the way how we are redacting objects using a new annotation.

kuzaxak requested a review from ilya-lesikov July 11, 2025 13:41

ilya-lesikov merged commit 7860bc2 into werf:main Jul 11, 2025
5 of 6 checks passed

ilya-lesikov added a commit that referenced this pull request Jul 11, 2025

feat: werf.io/sensitive-paths annotation and WERF_FEAT_FIELD_SENSITIV…

e3f9798

…E featgate (#364) Just to trigger release, the actual commit is 7860bc2 Signed-off-by: Ilya Lesikov <ilya@lesikov.com>

This was referenced Oct 23, 2025

chore(main): release 1.16.0 #446

Closed

chore(main): release 1.17.0 #451

Closed

This was referenced Nov 6, 2025

chore(main): release 1.17.0 #463

Closed

chore(main): release 1.18.0 #472

Closed

This was referenced Nov 13, 2025

chore(main): release 1.18.0 #481

Closed

chore(1.16): release 1.17.0 #486

Closed

chore(main): release 1.18.0 #487

Closed

This was referenced Dec 8, 2025

chore(main): release 1.19.0 #504

Merged

chore(main): release 1.20.0 #511

Closed

flant-team-sysdev mentioned this pull request Feb 4, 2026

chore(main): release 1.22.0 #559

Closed

flant-team-sysdev mentioned this pull request Feb 20, 2026

chore(main): release 1.22.1 #566

Open

Conversation

kuzaxak commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilya-lesikov commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jun 16, 2025

Uh oh!

ilya-lesikov commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilya-lesikov commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jun 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilya-lesikov commented Jul 4, 2025

Uh oh!

kuzaxak commented Jul 10, 2025

Uh oh!

ilya-lesikov left a comment

Choose a reason for hiding this comment

Uh oh!

kuzaxak commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuzaxak commented Jul 11, 2025

Uh oh!

ilya-lesikov commented Jul 11, 2025

Uh oh!

kuzaxak commented Jul 11, 2025

Uh oh!

Uh oh!

ilya-lesikov commented Jul 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kuzaxak commented Jun 14, 2025 •

edited

Loading

kuzaxak commented Jun 14, 2025 •

edited

Loading

ilya-lesikov commented Jun 16, 2025 •

edited

Loading

ilya-lesikov commented Jun 25, 2025 •

edited

Loading

kuzaxak commented Jun 25, 2025 •

edited

Loading

ilya-lesikov commented Jun 25, 2025 •

edited

Loading

kuzaxak commented Jun 25, 2025 •

edited

Loading

kuzaxak commented Jul 11, 2025 •

edited

Loading