Optional members in cell paths: Attempt 2 by rgwood · Pull Request #8379 · nushell/nushell

rgwood · 2023-03-10T02:15:46Z

This is a follow up from #7540. Please provide feedback if you have the time!

Summary

This PR lets you use ? to indicate that a member in a cell path is optional and Nushell should return null if that member cannot be accessed.

Unlike the previous PR, ? is now a postfix modifier for cell path members. A cell path of .foo?.bar means that foo is optional and bar is not.

? does not suppress all errors; it is intended to help in situations where data has "holes", i.e. the data types are correct but something is missing. Type mismatches (like trying to do a string path access on a date) will still fail.

Record Examples

{ foo: 123 }.foo # returns 123

{ foo: 123 }.bar # errors
{ foo: 123 }.bar? # returns null

{ foo: 123 } | get bar # errors
{ foo: 123 } | get bar? # returns null

{ foo: 123 }.bar.baz # errors
{ foo: 123 }.bar?.baz # errors because `baz` is not present on the result from `bar?`
{ foo: 123 }.bar.baz? # errors
{ foo: 123 }.bar?.baz? # returns null

List Examples

〉[{foo: 1} {foo: 2} {}].foo
Error: nu::shell::column_not_found

  × Cannot find column
   ╭─[entry #30:1:1]
 1 │ [{foo: 1} {foo: 2} {}].foo
   ·                    ─┬  ─┬─
   ·                     │   ╰── cannot find column 'foo'
   ·                     ╰── value originates here
   ╰────
〉[{foo: 1} {foo: 2} {}].foo?
╭───┬───╮
│ 0 │ 1 │
│ 1 │ 2 │
│ 2 │   │
╰───┴───╯
〉[{foo: 1} {foo: 2} {}].foo?.2 | describe
nothing

〉[a b c].4? | describe
nothing

〉[{foo: 1} {foo: 2} {}] | where foo? == 1
╭───┬─────╮
│ # │ foo │
├───┼─────┤
│ 0 │   1 │
╰───┴─────╯

Breaking changes

Column names with ? in them now need to be quoted.
The -i/--ignore-errors flag has been removed from get and select
1. After this PR, most get error handling can be done with ? and/or try/catch.
Cell path accesses like this no longer work without a ?:

〉[{a:1 b:2} {a:3}].b.0
2

We had some clever code that was able to recognize that since we only want row 0, it's OK if other rows are missing column b. I removed that because it's tricky to maintain, and now that query needs to be written like:

〉[{a:1 b:2} {a:3}].b?.0
2

I think the regression is acceptable for now. I plan to do more work in the future to enable streaming of cell path accesses, and when that happens I'll be able to make .b.0 work again.

codecov · 2023-03-10T02:38:19Z

Codecov Report

Merging #8379 (8c7f501) into main (aa876ce) will decrease coverage by 0.03%.
The diff coverage is 69.91%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8379      +/-   ##
==========================================
- Coverage   68.06%   68.04%   -0.03%     
==========================================
  Files         621      621              
  Lines       99885    99943      +58     
==========================================
+ Hits        67988    68002      +14     
- Misses      31897    31941      +44

Impacted Files	Coverage Δ
crates/nu-command/src/debug/inspect_table.rs	`0.00% <0.00%> (ø)`
crates/nu-explore/src/nu_common/table.rs	`0.00% <0.00%> (ø)`
crates/nu-explore/src/nu_common/value.rs	`0.00% <0.00%> (ø)`
crates/nu-parser/src/parse_keywords.rs	`68.51% <0.00%> (-0.05%)`	⬇️
crates/nu-command/src/filters/drop/column.rs	`80.00% <33.33%> (+2.15%)`	⬆️
crates/nu-protocol/src/ast/cell_path.rs	`53.19% <35.71%> (-23.00%)`	⬇️
crates/nu-protocol/src/value/from_value.rs	`34.97% <50.00%> (+0.07%)`	⬆️
crates/nu-protocol/src/value/mod.rs	`56.29% <59.64%> (-0.28%)`	⬇️
crates/nu-cli/src/repl.rs	`24.35% <88.57%> (ø)`
crates/nu-parser/src/parser.rs	`83.26% <91.52%> (+0.05%)`	⬆️
... and 15 more

... and 1 file with indirect coverage changes

fdncred · 2023-03-10T12:42:33Z

Just to be clear, the reason you're suggesting we remove --ignore-errors/-i is because:

These produce identical results

{ foo: 123 } | get bar?
{ foo: 123 } | get -i bar

and these produce identical results

[{a:1 b:2} {a:3}] | select b?
[{a:1 b:2} {a:3}] | select -i b

If that is correct, then I'd propose dropping the -i in this PR, and maybe have a deprecate message on -i?

rgwood · 2023-03-10T16:25:33Z

Right, ? can replace -i in most situations. For others, try/catch can be used. I’ll remove -i

rgwood · 2023-03-10T17:14:12Z

On second thought... removing -i is going to be hard, it's used in default_env.nu 😞

let home = ($env | get -i (if $nu.os-info.name == "windows" { "USERPROFILE" } else { "HOME" }) | into string)

Removing the flag will break people's env.nu. Maybe we should leave it around for now.

rgwood · 2023-03-10T17:39:12Z

Oh! I just noticed that the get -i was added to default_env.nu after the 0.76 release: #8173

If I remove it before the 0.77 release, that will make future deprecation of -i much easier. Will do that in another PR.

This is a follow-up from #8173, which was merged shortly after the 0.76 release. That PR changed `default_env.nu` so that the user's home folder is displayed as `~` in the left prompt. It did so using `get -i`. This PR just rewrites the Nu code from #8173 to use `try`/`catch` instead of `-i`, which will make it easier to remove the `-i` flags from `get` and `select` eventually (see #8379). I would like to merge this before the 0.77 release, so we don't end up with lots of `env.nu` files using `get -i` out in the wild.

rgwood · 2023-03-10T19:23:41Z

I've removed the -i flag from get and select. Hopefully that won't make too many people angry.

webbedspace · 2023-03-11T09:37:35Z

Thanks for revisiting this feature!

rgwood · 2023-03-16T03:50:52Z

I asked in Discord and several people voted to merge this. Let's give it a try!

#8379 removed the `-i` flag from `get` and `select` because the new `?` functionality covers most of the same use cases. However, #8480 made me realize that `-i` is still useful when dealing with cell paths in variables. This PR re-adds the `-i` flag to `get` and `select`. It works by just marking every member in the cell path as optional, which will behave _slightly_ differently than `-i` used to (previously it would suppress any errors, even type errors) but IMO that's OK.

This is a follow-up to #8379 and #8502. This PR makes it so that the new `?` syntax for marking a path member as optional short-circuits, as voted on in the [8502](#8502) poll. Previously, `{ foo: 123 }.bar?.baz` would raise an error: ``` > { foo: 123 }.bar?.baz × Data cannot be accessed with a cell path ╭─[entry #15:1:1] 1 │ { foo: 123 }.bar?.baz · ─┬─ · ╰── nothing doesn't support cell paths ╰──── ``` Here's what was happening: 1. The `bar?` path member access returns `nothing` because there is no field named `bar` on the record 2. The `baz` path member access fails when trying to access a `baz` field on that `nothing` value After this change, `{ foo: 123 }.bar?.baz` returns `nothing`; the failed `bar?` access immediately returns `nothing` and the `baz` access never runs.

rgwood added 11 commits March 8, 2023 20:55

create cell path tests

685aa36

just get it building

ca18019

Add cell path parser tests

fa2a809

fix TODOs

4fb6643

Get parser working with ?

87fecf6

change short circuit test

a3775eb

add more tests

b8c6b4c

Down to 3 test failures

398ee07

nearly all tests passing

819e15e

fix final test

afce069

clippy

5c22ca0

rgwood marked this pull request as draft March 10, 2023 02:15

rgwood mentioned this pull request Mar 10, 2023

Remove get -i from default env file #8390

Merged

rgwood added 2 commits March 10, 2023 11:09

Merge branch 'main' into optional-chaining-attempt-2

658d62e

Remove the -i flag from get and select

1421879

rgwood mentioned this pull request Mar 10, 2023

Optional chaining for cell paths (?.) #7540

Closed

rgwood marked this pull request as ready for review March 14, 2023 16:39

rgwood added 2 commits March 15, 2023 15:04

Merge branch 'main' into optional-chaining-attempt-2

b383446

fix up merge conflict

8c7f501

rgwood merged commit 21b84a6 into nushell:main Mar 16, 2023

rgwood added the notes:breaking-changes This PR implies a change affecting users and has to be noted in the release notes label Mar 16, 2023

ghost mentioned this pull request Mar 16, 2023

New optional cell-path make use of variables harder #8480

Closed

rgwood mentioned this pull request Mar 16, 2023

Add -i flag back to get and select #8488

Merged

rgwood mentioned this pull request Mar 21, 2023

Cell paths: make optional path members short-circuit #8554

Merged

rgwood mentioned this pull request Mar 25, 2023

Work easily with irregular (json) data as tables #7655

Closed

mon-jai mentioned this pull request Apr 5, 2023

doubt: why is the access API of record designed in this way? #8746

Closed

amtoine mentioned this pull request May 29, 2023

feat: update the nushell init file and make it valid module and overlay starship/starship#5188

Merged

5 tasks

coderabbitai bot mentioned this pull request Mar 21, 2026

feat: Ctx Enhancements - Edinburgh Protocol Tools, Mentation Workflow & Entropy Scoring pjsvis/mastra-hono#9

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optional members in cell paths: Attempt 2#8379

Optional members in cell paths: Attempt 2#8379
rgwood merged 15 commits intonushell:mainfrom
rgwood:optional-chaining-attempt-2

rgwood commented Mar 10, 2023 •

edited

Loading

Uh oh!

codecov bot commented Mar 10, 2023 •

edited

Loading

Uh oh!

fdncred commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023 •

edited

Loading

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

webbedspace commented Mar 11, 2023

Uh oh!

rgwood commented Mar 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rgwood commented Mar 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Record Examples

List Examples

Breaking changes

Uh oh!

codecov bot commented Mar 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fdncred commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

rgwood commented Mar 10, 2023

Uh oh!

webbedspace commented Mar 11, 2023

Uh oh!

rgwood commented Mar 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rgwood commented Mar 10, 2023 •

edited

Loading

codecov bot commented Mar 10, 2023 •

edited

Loading

rgwood commented Mar 10, 2023 •

edited

Loading