Skip to content

feat: Add SensitiveColumns to tables schema#2134

Merged
kodiakhq[bot] merged 7 commits intomainfrom
feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk
May 9, 2025
Merged

feat: Add SensitiveColumns to tables schema#2134
kodiakhq[bot] merged 7 commits intomainfrom
feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk

Conversation

@blesniewski
Copy link
Copy Markdown
Contributor


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@github-actions github-actions bot added the feat label Apr 29, 2025
@blesniewski
Copy link
Copy Markdown
Contributor Author

Needs a cloudquery-api-go merge first

@murarustefaan
Copy link
Copy Markdown
Member

Does this work with:

  • nested fields? JSONPath, basically? we should probably have a solution for that as well (say .spec.containers[0].env in the kubernetes_pods table). If not doable, than we can retrhink the approach of using the transformer and do table_options 🤔
  • can we somehow set these in the arrow schema as well? so the transformer would know of all the sensitive columns and redact them by default with no configuration whatsoever?

@blesniewski
Copy link
Copy Markdown
Contributor Author

blesniewski commented May 2, 2025

nested fields? JSONPath

This is analogical to PermissionsNeeded, meaning it could contain whatever strings we need for each table, then it's on the transformer side to handle the more complicated cases, which I believe it does as per the docs https://hub.cloudquery.io/plugins/transformer/cloudquery/basic/latest/docs

can we somehow set these in the arrow schema as well?

This we'd have to investigate

Edit:
From my local tests, everything should work as above, meaning we should be able to read from the arrow schema in the basic transformer, and basing on that redact what's marked on table definition

@blesniewski blesniewski marked this pull request as ready for review May 7, 2025 15:44
@blesniewski blesniewski requested review from a team and przste-go May 7, 2025 15:44
@kodiakhq kodiakhq bot merged commit e95674f into main May 9, 2025
10 checks passed
@kodiakhq kodiakhq bot deleted the feature/eng-1033-allow-specifying-sensitive-table-columns-in-the-sdk branch May 9, 2025 10:18
kodiakhq bot pushed a commit that referenced this pull request May 9, 2025
🤖 I have created a release *beep* *boop*
---


## [4.80.0](v4.79.1...v4.80.0) (2025-05-09)


### Features

* Add SensitiveColumns to tables schema ([#2134](#2134)) ([e95674f](e95674f))


### Bug Fixes

* **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.13.9 ([#2143](#2143)) ([77d4b9b](77d4b9b))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request May 15, 2025
#### Summary

Adds automatic obfuscation transformation basing on the new metadata field added in cloudquery/plugin-sdk#2134
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants