Skip to content

feat: Add 'auto_obfuscate' transformation to basic transformer#20728

Merged
kodiakhq[bot] merged 7 commits intomainfrom
feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk
May 15, 2025
Merged

feat: Add 'auto_obfuscate' transformation to basic transformer#20728
kodiakhq[bot] merged 7 commits intomainfrom
feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk

Conversation

@blesniewski
Copy link
Copy Markdown
Contributor

Summary

Adds automatic obfuscation transformation basing on the new metadata field added in cloudquery/plugin-sdk#2134

@cq-bot cq-bot added the area/cli label May 8, 2025
@blesniewski
Copy link
Copy Markdown
Contributor Author

blesniewski commented May 8, 2025

Not ready yet:

  • Needs the SDK PR
  • docs need to be updated

Also, not sure if:

  • we should add way to skip certain tables and to set that in spec
  • modifying the default output of obfuscate_columns is ok, or if this should have it's own logic to do the obfuscation

…feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk
@blesniewski
Copy link
Copy Markdown
Contributor Author

blesniewski commented May 9, 2025

One question remaining:
If there's a JSON column, the current obfuscate transformation behavior is that it won't obfuscate the entire column- it requires a json path: failed to transform schema: column tags is not a string column

So 2 options:

  • if we want this capability, I'll probably split the logic between obfuscate and auto_obfuscate, and add handling (or adjust the obfuscate behavior if we want to keep sharing)
  • if we don't want to be able to obfuscate entire json columns, I'll adjust the validation in SDK

I assume we do want it, just putting the question out for confirmation @murarustefaan

@blesniewski
Copy link
Copy Markdown
Contributor Author

I've added handling for obfuscating entire JSON columns.

Drawback of the current solution - to not have to unmarshal every value for each handled column, we're calculating hash of the entire value, which results in JSON output, but without preserving the internal structure:
{"redacted_by_cloudquery": "81f2a9ddc7ae49a...hash_value"}

Additionally, the output of the obfuscate transformation has changed, now the hashes values in would be preceded by Redacted by CloudQuery |

@blesniewski blesniewski marked this pull request as ready for review May 9, 2025 14:54
@blesniewski blesniewski requested review from a team and maaarcelino May 9, 2025 14:54
@blesniewski blesniewski added the automerge Automatically merge once required checks pass label May 15, 2025
@kodiakhq kodiakhq bot merged commit 66cbf0a into main May 15, 2025
19 checks passed
@kodiakhq kodiakhq bot deleted the feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk branch May 15, 2025 08:36
kodiakhq bot pushed a commit that referenced this pull request May 15, 2025
🤖 I have created a release *beep* *boop*
---


## [2.3.0](plugins-transformer-basic-v2.2.5...plugins-transformer-basic-v2.3.0) (2025-05-15)


### Features

* Add 'auto_obfuscate' transformation to basic transformer ([#20728](#20728)) ([66cbf0a](66cbf0a))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq bot pushed a commit that referenced this pull request May 19, 2025
🤖 I have created a release *beep* *boop*
---


## [6.20.0](cli-v6.19.2...cli-v6.20.0) (2025-05-19)


### Features

* Add 'auto_obfuscate' transformation to basic transformer ([#20728](#20728)) ([66cbf0a](66cbf0a))


### Bug Fixes

* **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.80.2 ([#20778](#20778)) ([525352c](525352c))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants