Skip to content

CLI: Refactor BoxRenderer, add wrapping support for large values, and add pretty printing / highlighting for nested types, JSON and variant#19721

Merged
Mytherin merged 36 commits intoduckdb:mainfrom
Mytherin:boxrendererlargevalue
Nov 11, 2025
Merged

Conversation

@Mytherin
Copy link
Collaborator

@Mytherin Mytherin commented Nov 10, 2025

This PR does a code refactor of the BoxRenderer - cleaning up the code by splitting up the generating of values to render from the actual rendering.

Wrapping

In addition, this PR adds support for wrapping for large values. The way this works is that, when we select fewer rows than the .maxrows, we allow the result rendering to use multiple lines to display a single row in the query result. Larger values can then be stretched out over multiple lines instead of being truncated, allowing them to be visible. For example:

select 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.' as lorem_ipsum;
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                          lorem_ipsum                                           │
│                                            varchar                                             │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut   │
│ labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco lab  │
│ oris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in volup  │
│ tate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non  │
│  proident, sunt in culpa qui officia deserunt mollit anim id est laborum.                      │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

If we exceed the maxrows, we don't wrap, but still truncate:

.maxrows 10
select 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.' as lorem_ipsum
from range(100);
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                          lorem_ipsum                                           │
│                                            varchar                                             │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│                                               ·                                                │
│                                               ·                                                │
│                                               ·                                                │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut…  │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                      100 rows (10 shown)                                       │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

Similarly, we only allow wrapping until the maxrows is reached. If we have a gigantic string we will be able to fit more of it in the result, but still not everything:

select repeat('abcdefghijklmnopqrstuvwxyz', 10000) r;
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                               r                                                │
│                                            varchar                                             │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmno  │
│ pqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcd  │
│ efghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs  │
│ tuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefgh  │
│ ijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw  │
│ xyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijkl  │
│ mnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz…  │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

If we select multiple large strings, we might be able to do some wrapping - but we never exceed .maxrows. The wrapping is done greedily (i.e. we try to add wrapping for the first value, then the second one, etc). For example:

memory D select 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.' as lorem_ipsum
from range(2);
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                       lorem_ipsum                                       │
│                                         varchar                                         │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incidid  │
│ unt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitati  │
│ on ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in r  │
│ eprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur  │
│  sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim…  │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incidi…  │
└─────────────────────────────────────────────────────────────────────────────────────────┘

Nested types / JSON / VARIANT

When doing this wrapping for nested types, JSON or variant, we add a "pretty print" step that tries to format the JSON in a nice way - provided it fits on the screen. For example:

file.json
{
  "aliceblue": "#f0f8ff",
  "antiquewhite": "#faebd7",
  "aqua": "#00ffff",
  "aquamarine": "#7fffd4",
  "azure": "#f0ffff",
  "beige": "#f5f5dc",
  "bisque": "#ffe4c4",
  "black": "#000000",
  "blanchedalmond": "#ffebcd",
  "blue": "#0000ff",
  "blueviolet": "#8a2be2",
  "brown": "#a52a2a",
}
from read_json_objects('file.json');
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                              json                                              │
│                                              json                                              │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {                                                                                              │
│   "aliceblue": "#f0f8ff",                                                                      │
│   "antiquewhite": "#faebd7",                                                                   │
│   "aqua": "#00ffff",                                                                           │
│   "aquamarine": "#7fffd4",                                                                     │
│   "azure": "#f0ffff",                                                                          │
│   "beige": "#f5f5dc",                                                                          │
│   "bisque": "#ffe4c4",                                                                         │
│   "black": "#000000",                                                                          │
│   "blanchedalmond": "#ffebcd",                                                                 │
│   "blue": "#0000ff",                                                                           │
│   "blueviolet": "#8a2be2",                                                                     │
│   "brown": "#a52a2a"                                                                           │
│ }                                                                                              │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

The formatter will try to adapt to the max width of the terminal, and use a more compact rendering if the terminal width is exceeded by the simpler one. For example:

select json_serialize_sql($$select '10'::blob$$);
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                          json_serialize_sql('select ''10''::blob')                                          │
│                                                            json                                                             │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {                                                                                                                           │
│   "error": false,                                                                                                           │
│   "statements": [                                                                                                           │
│     {                                                                                                                       │
│       "node": {                                                                                                             │
│         "type": "SELECT_NODE",                                                                                              │
│         "modifiers": [],                                                                                                    │
│         "cte_map": {"map": []},                                                                                             │
│         "select_list": [                                                                                                    │
│           {                                                                                                                 │
│             "class": "CONSTANT",                                                                                            │
│             "type": "VALUE_CONSTANT",                                                                                       │
│             "alias": "",                                                                                                    │
│             "query_location": 18446744073709551615,                                                                         │
│             "value": {"type": {"id": "BLOB", "type_info": null}, "is_null": false, "value": "10"}                           │
│           }                                                                                                                 │
│         ],                                                                                                                  │
│         "from_table": {"type": "EMPTY", "alias": "", "sample": null, "query_location": 18446744073709551615},               │
│         "where_clause": null,                                                                                               │
│         "group_expressions": [],                                                                                            │
│         "group_sets": [],                                                                                                   │
│         "aggregate_handling": "STANDARD_HANDLING",                                                                          │
│         "having": null,                                                                                                     │
│         "sample": null,                                                                                                     │
│         "qualify": null                                                                                                     │
│       },                                                                                                                    │
│       "named_param_map": []                                                                                                 │
│     }                                                                                                                       │
│   ]                                                                                                                         │
│ }                                                                                                                           │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                        json_serialize_sql('select ''10''::blob')                         │
│                                           json                                           │
├──────────────────────────────────────────────────────────────────────────────────────────┤
│ {                                                                                        │
│   "error": false,                                                                        │
│   "statements": [{                                                                       │
│       "node": {                                                                          │
│         "type": "SELECT_NODE", "modifiers": [], "cte_map": {"map": []},                  │
│         "select_list": [{                                                                │
│             "class": "CONSTANT", "type": "VALUE_CONSTANT", "alias": "",                  │
│             "query_location": 18446744073709551615,                                      │
│             "value": {                                                                   │
│               "type": {"id": "BLOB", "type_info": null}, "is_null": false,               │
│               "value": "10"                                                              │
│             }                                                                            │
│           }                                                                              │
│         ],                                                                               │
│         "from_table": {                                                                  │
│           "type": "EMPTY", "alias": "", "sample": null,                                  │
│           "query_location": 18446744073709551615                                         │
│         }, "where_clause": null, "group_expressions": [], "group_sets": [],              │
│         "aggregate_handling": "STANDARD_HANDLING", "having": null, "sample": null,       │
│         "qualify": null                                                                  │
│       }, "named_param_map": []                                                           │
│     }                                                                                    │
│   ]                                                                                      │
│ }                                                                                        │
└──────────────────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│                  json_serialize_sql('select ''10''::blob')                   │
│                                     json                                     │
├──────────────────────────────────────────────────────────────────────────────┤
│ {                                                                            │
│   "error": false,                                                            │
│   "statements": [{                                                           │
│       "node": {                                                              │
│         "type": "SELECT_NODE", "modifiers": [], "cte_map": {"map": []},      │
│         "select_list": [{                                                    │
│             "class": "CONSTANT", "type": "VALUE_CONSTANT", "alias": "",      │
│             "query_location": 18446744073709551615,                          │
│             "value": {                                                       │
│               "type": {"id": "BLOB", "type_info": null}, "is_null": false,   │
│               "value": "10"                                                  │
│             }                                                                │
│           }                                                                  │
│         ],                                                                   │
│         "from_table": {                                                      │
│           "type": "EMPTY", "alias": "", "sample": null,                      │
│           "query_location": 18446744073709551615                             │
│         }, "where_clause": null, "group_expressions": [], "group_sets": [],  │
│         "aggregate_handling": "STANDARD_HANDLING", "having": null,           │
│         "sample": null, "qualify": null                                      │
│       }, "named_param_map": []                                               │
│     }                                                                        │
│   ]                                                                          │
│ }                                                                            │
└──────────────────────────────────────────────────────────────────────────────┘

Just like with wrapping, this formatting is only done if there is space for it, e.g.:

select json_serialize_sql($$select '10'::blob$$) from range(100);
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                       json_serialize_sql('select ''10''::blob')                                       │
│                                                         json                                                          │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│                                                           ·                                                           │
│                                                           ·                                                           │
│                                                           ·                                                           │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
│ {"error":false,"statements":[{"node":{"type":"SELECT_NODE","modifiers":[],"cte_map":{"map":[]},"select_list":[{"cla…  │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                  100 rows (10 shown)                                                  │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

More space can always be made by the user by running e.g. .maxrows -1.

Highlighting

We also add highlighting support for JSON / VARIANT / nested types in results, e.g.:

Screenshot 2025-11-10 at 13 03 26

@carlopi
Copy link
Contributor

carlopi commented Nov 10, 2025

Mostly curiosity: can / is the JSON rendering applied also for STRUCTs and/or MAPs? Or maybe even ARRAY/LISTs?

@Mytherin
Copy link
Collaborator Author

Should work with all nested types yes

@Mytherin Mytherin merged commit c95fd26 into duckdb:main Nov 11, 2025
90 of 135 checks passed
github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request Nov 11, 2025
Refactor BoxRenderer, add wrapping support for large values, and add pretty printing / highlighting for nested types, JSON and variant (duckdb/duckdb#19721)
Internal duckdb/duckdb#6523: AsOf Exception Handling (duckdb/duckdb#19703)
github-actions bot added a commit to duckdb/duckdb-r that referenced this pull request Nov 11, 2025
Refactor BoxRenderer, add wrapping support for large values, and add pretty printing / highlighting for nested types, JSON and variant (duckdb/duckdb#19721)
Internal duckdb/duckdb#6523: AsOf Exception Handling (duckdb/duckdb#19703)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>
@Mytherin Mytherin changed the title Refactor BoxRenderer, add wrapping support for large values, and add pretty printing / highlighting for nested types, JSON and variant CLI: Refactor BoxRenderer, add wrapping support for large values, and add pretty printing / highlighting for nested types, JSON and variant Nov 15, 2025
Mytherin added a commit that referenced this pull request Nov 16, 2025
… array entry unless there are complex objects in the array (#19795)

Follow-up fix from #19721

When rendering an array that does not contain any complex objects, wrap
it on multiple lines instead of adding newlines after every comma, e.g.
we render it as this:

```
[1, 2, 3, 4,
 5, 6, 7, 8]
```

Instead of this:

```
[
   1,
   2,
   3,
   4,
   5,
   6,
   7,
   8
]
```
@Mytherin Mytherin deleted the boxrendererlargevalue branch December 4, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants