Fix roundtripping of stringified nested types#16304
Merged
Mytherin merged 63 commits intoduckdb:mainfrom Mar 19, 2025
Merged
Conversation
…g and trailing spaces
…will already have escaped whatever was necessary to be escaped
… special characters
… LIST and STRUCT, add escape support as well
…, only escaping singlequotes and backslashes within that string
Collaborator
|
Thanks! |
hmeriann
added a commit
to hmeriann/duckdb
that referenced
this pull request
Mar 21, 2025
Mytherin
added a commit
that referenced
this pull request
Mar 21, 2025
…roundtripping of stringified nested types` PR (#16775) This PR updates the expected results according to the changes merged with [this PR](#16304), which should resolve the issue: duckdblabs/duckdb-internal#4477
2 tasks
2 tasks
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 15, 2025
Fix roundtripping of stringified nested types (duckdb/duckdb#16304)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 15, 2025
Fix roundtripping of stringified nested types (duckdb/duckdb#16304)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 16, 2025
Fix roundtripping of stringified nested types (duckdb/duckdb#16304)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 16, 2025
Fix roundtripping of stringified nested types (duckdb/duckdb#16304)
krlmlr
added a commit
to duckdb/duckdb-r
that referenced
this pull request
May 17, 2025
Fix roundtripping of stringified nested types (duckdb/duckdb#16304)
hannes
added a commit
that referenced
this pull request
Aug 11, 2025
…8541) and use the `ExtensionLoader` to register functions instead of the old API. PR #16304 fixed some `LIST` casting issues regarding round-tripping (e.g., adding quotes or escaping them), which introduced casting issues for JSON. These issues were found by the JSON tests, but the tests were simply changed to match the new behavior instead of fixing the issues. The issue seemed like a rendering issue only, but when returning `JSON` to our clients, e.g., Python, we cast to `VARCHAR`, which now adds quotes or escapes them, creating invalid JSON. This PR fixes these issues by creating special-case casts from `VARCHAR` to `JSON` and vice versa. I don't think this is needed for structs, as the default cast (which is to reinterpret) is used for structs, and we shouldn't get invalid JSON (e.g., surrounded by single quotes) when getting a struct that contains JSON into a client like Python. Fixes: * #17647 * duckdblabs/duckdb-internal#5498
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes https://github.com/duckdblabs/duckdb-internal/issues/3945
Follow up to #15944
Single quotes are added around the value when it contains special characters, trailing/leading whitespace or the value is "NULL".
If single quotes or backslashes are present in the value, they are escaped.
Added benchmark results:
New (this PR):
Old (
main):(casting from map used to be done by appending to a
std::string, which explains the speed up)Misc
generate_extension_entries.pyreceived some changes.Most notably the deserializing of the generated
extension_entries.hpparray entries has been changed from a regex approach to manually parsing the input (to address an unrelated issue, related to entries that span multiple lines).We used to query the shell with a
-csvflag, expecting this to produce unquoted values, which is no longer the case as we need to escapeVARCHAR[]among others.This has been changed to use
-jsoninstead, so we can use thejsonpython library to reliably retrieve the results.In
built_in_functions.cpp,BuiltinFunctions::RegisterExtensionOverloadsnow parses the list of parameter types by casting toVARCHAR[]then iterating over the produced list entries.