Skip to content

fix: Ignore utf8 keys.#19643

Merged
marianogappa merged 1 commit intomainfrom
mariano/ignore-utf8-keys
Nov 20, 2024
Merged

fix: Ignore utf8 keys.#19643
marianogappa merged 1 commit intomainfrom
mariano/ignore-utf8-keys

Conversation

@marianogappa
Copy link
Copy Markdown
Contributor

@marianogappa marianogappa commented Nov 20, 2024

fixes https://github.com/cloudquery/cloudquery-issues/issues/2763
(internal issue)

Normally, the JSON Flattener will attempt to flatten JSON objects to the first level, e.g.:

"type_schema": "[{\"availability_domain\":\"utf8\",\"display_name\":\"utf8\",\"volume_group_replica_id\":\"utf8\"}]"

But the TypeSchema notation had a special case that the JSON Flattener wasn't aware of:

      {
        "name": "freeform_tags",
        "type": "json",
        "type_schema": "{\"utf8\":\"utf8\"}"
      },

In this case, the utf8 key is not a literal string, but a special value that means "any string key". This should be ignored by the flattener, because it's meant to contain different keys on different rows, and table schemas are fixed.

The only way to implement this would be to do a linear pass over all rows and learn all possible keys, then do another pass and create nullable columns on all non-supplied columns for each row. We definitely don't want to support this.

Instead, this PR simply ignores keys with this special literal string.

The practice is common on the Oracle source plugin; here's a flattened Oracle table that was suffering from this issue, now fixed:

Screenshot 2024-11-20 at 10 14 43

@marianogappa marianogappa changed the title fix: ignore utf8 keys. fix: Ignore utf8 keys. Nov 20, 2024
@marianogappa marianogappa marked this pull request as ready for review November 20, 2024 06:24
@marianogappa marianogappa requested review from a team and savme and removed request for a team November 20, 2024 06:24
@marianogappa marianogappa merged commit 411474a into main Nov 20, 2024
@marianogappa marianogappa deleted the mariano/ignore-utf8-keys branch November 20, 2024 07:59
kodiakhq bot pushed a commit that referenced this pull request Nov 20, 2024
🤖 I have created a release *beep* *boop*
---


## [1.0.3](plugins-transformer-jsonflattener-v1.0.2...plugins-transformer-jsonflattener-v1.0.3) (2024-11-20)


### Bug Fixes

* **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.68.3 ([#19604](#19604)) ([3d378ea](3d378ea))
* Ignore utf8 keys. ([#19643](#19643)) ([411474a](411474a))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants