stdlib: add an automatic loading of "prelude" commands by amtoine · Pull Request #8627 · nushell/nushell

amtoine · 2023-03-26T11:12:20Z

Should address some points in #8450.
Related to #8611.

Description

parse the std.nu module and add
- the library as std to the main "namespace"
- the top-level comments of the library to have a complete help std
- all the commands in the library to the useable set of commands with use
loads the prelude, i.e. a list of (name, search_name) to be added automatically
- name is the name of the command that the user can use without use
- search_name is the name of the command in the std.nu module
- most of the time, name and search_name will be the same, but the name can be made std ... to make the difference with built-in commands
finally merges the loaded library and imported prelude to the main engine state
throws errors at each parsing / adding step without crashing the shell

🧪 tests have been added to test the bad behaviours and bug i have stumbled upon and the expected behaviour.

User-Facing Changes

normal behaviour

the user will have access, for free, to the commands listed in the prelude, e.g. the help implementations in nushell from #8611.

the standard library will be available, for free, as std.
the user can import any command from the standard library with

use std <command>

or the whole library with

use std

from anywhere, without having to have the std.nu file locally on disk.

when the prelude can not be loaded

Error:
  × Unable to load the prelude of the standard library.
  help: this is a bug: please file an issue in the [issue tracker](https://github.com/nushell/nushell/issues/new/choose)

Error:
  × could not load `not an std command` from `std`.

Error:
  × could not load `not an valid command either` from `std`.


     __  ,
 .--()°'.' Welcome to Nushell,
'|, . ,'   based on the nu language,
 !_-(_\    where all data is structured!

Please join our Discord community at https://discord.gg/NtAbbGn
Our GitHub repository is at https://github.com/nushell/nushell
Our Documentation is located at https://nushell.sh
Tweet us at @nu_shell
Learn how to remove this at: https://nushell.sh/book/configuration.html#remove-welcome-message

It's been this long since Nushell's first commit:
3yr 10month 3wk 3day 15hr 8min 368ms 488µs 475ns

Startup Time: 129ms 949µs 70ns

Note
see how it does NOT crash nushell, simply does not load these commands 👌

Tests + Formatting

🟢 toolkit fmt
🟢 toolkit clippy
🟢 toolkit test

After Submitting

this will make the annoucement of the standard library possible!
we will be able to land #8406 and #8415 and #8611 when we want 👌

amtoine · 2023-03-29T16:53:26Z

i force-pushed because there was nothing to keep from b037038 to 0d05fd4...

now the changes to look at are between 525f66a and the tip of the branch 👍

amtoine · 2023-03-29T16:59:27Z

thanks to @kubouch, it's almost working

std is made available during the prelude
we can use commands from std without having them on the disk

Note
see the original comments of @kubouch here and here

however, i have an error when trying to run a command imported from std 🤔

reproduce the 👍 and the 🐛

remove the std.nu of the nushell source from NU_LIB_DIRS to avoid cheating
run

cargo run -- --no-config-file

run help assert => it works 🟢
run help assert equal => it does not work 🟢
run use std "assert equal" => it works 🟢
run help assert equal => now it works 🟢
try to write assert to run the command => nushell panics 🔴

thread 'main' panicked at 'Attempt to mutate a block that is in the permanent (immutable) state', crates/nu-protocol/src/engine/engine_state.rs:2055:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

❗ TL;DR: almost there 😌

kubouch · 2023-03-29T18:34:23Z

src/run.rs

+        working_set.add_file(name.clone(), content);
+        let end = working_set.next_span_start();
+
+        let (_, module, _, _) = parse_module_block(


Here it would be good to preserve the module comments and pass them to add_module() so you can have a nice help message when calling help std.

should be good in 61436bc, help std gives the comment now

kubouch · 2023-03-29T18:36:02Z

src/run.rs

+                            .decls
+                            .get(&search_name.as_bytes().to_vec())
+                            .unwrap_or_else(|| {
+                                panic!("could not load `{}` from `std`.", search_name)


We can't panic here, it should probably print an error and continue running. There are report_error() and report_error_new() functions for printing the error.

it does not appear to solve the issue, but still in d0101dd 👌

kubouch · 2023-03-29T18:41:08Z

src/run.rs

+        working_set.add_file(name.clone(), content);
+        let end = working_set.next_span_start();
+
+        let (_, module, _, _) = parse_module_block(


Oh, and the panic you're seeing is probably caused by missing working_set.add_block(). You'll need to call that and pass the block returned from here to it. You can see an example in use implementation code:

nushell/crates/nu-parser/src/parse_keywords.rs

Line 2269 in a49e5b3

let _ = working_set.add_block(block);

i've tried something in fccbcb0, see my comment at the top where i explain how it behaves now 👍

kubouch · 2023-03-29T18:43:04Z

Looks good, almost there. One thing I would change is to load std not just in REPL, but also when running commands/files. To me, it makes sense to have std available everywhere.

kubouch · 2023-03-29T18:44:49Z

src/run.rs

+        working_set.add_file(name.clone(), content);
+        let end = working_set.next_span_start();
+
+        let (_, module, _, _) = parse_module_block(


Also (sorry for the spam :-D ), the error returned from parsing the module should be printed, if there is any. See the comment below how to report errors.

there it is 😋 a503868

kubouch · 2023-03-29T21:07:36Z

@amtoine Could you paste a backtrace from the panic? It's strange that it needs a mutable block...

amtoine · 2023-03-30T16:48:46Z

Looks good, almost there. One thing I would change is to load std not just in REPL, but also when running commands/files. To me, it makes sense to have std available everywhere.

done in 3d01588 and 6f239f9 😉

amtoine · 2023-03-30T16:50:16Z

@amtoine Could you paste a backtrace from the panic? It's strange that it needs a mutable block...

of course, there it is

>_ RUST_BACKTRACE=full cargo run -- -c "assert"
thread 'main' panicked at 'Attempt to mutate a block that is in the permanent (immutable) state', crates/nu-protocol/src/engine/engine_state.rs:2055:13
stack backtrace:
   0:     0x562dc41ecca0 - std::backtrace_rs::backtrace::libunwind::trace::h1d00f3fcf4cb5ac4
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x562dc41ecca0 - std::backtrace_rs::backtrace::trace_unsynchronized::h920a6ff332484ee2
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x562dc41ecca0 - std::sys_common::backtrace::_print_fmt::hd7323920c925af6d
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x562dc41ecca0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h3155a8c966b4beb5
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x562dc4217c1e - core::fmt::write::h062c617411b691df
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/fmt/mod.rs:1209:17
   5:     0x562dc41e4c85 - std::io::Write::write_fmt::hb61fdf1275c61e1c
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/io/mod.rs:1682:15
   6:     0x562dc41eca65 - std::sys_common::backtrace::_print::hd1b4d9664ab500e0
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x562dc41eca65 - std::sys_common::backtrace::print::hca896ae22beb06cb
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x562dc41ee92f - std::panicking::default_hook::{{closure}}::h0b5eeed5cf36ab5f
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:267:22
   9:     0x562dc41ee66a - std::panicking::default_hook::h8932b573145a321b
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:286:9
  10:     0x562dc14259c0 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h420bcb63e1514d33
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/alloc/src/boxed.rs:2001:9
  11:     0x562dc142d79a - nu::main::{{closure}}::ha942bc60a773a159
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/src/main.rs:39:9
  12:     0x562dc41ef049 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hbb63890df2a28e48
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/alloc/src/boxed.rs:2001:9
  13:     0x562dc41ef049 - std::panicking::rust_panic_with_hook::h4b1447a24e3e94f8
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:692:13
  14:     0x562dc41eed81 - std::panicking::begin_panic_handler::{{closure}}::h8701da9995a3820c
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:577:13
  15:     0x562dc41ed14c - std::sys_common::backtrace::__rust_end_short_backtrace::hb696c5ed02a01598
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:137:18
  16:     0x562dc41eeae2 - rust_begin_unwind
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:575:5
  17:     0x562dc1419b93 - core::panicking::panic_fmt::h8aa706a976963c88
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/panicking.rs:65:14
  18:     0x562dc3cf7436 - nu_protocol::engine::engine_state::StateWorkingSet::get_block_mut::h86bdc8ee64c13065
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/crates/nu-protocol/src/engine/engine_state.rs:2055:13
  19:     0x562dc2d19423 - nu_parser::parser::parse::h0783cf7d1fc1f8a6
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/crates/nu-parser/src/parser.rs:6639:25
  20:     0x562dc1482f05 - nu_cli::commands::evaluate_commands::hb98951cbe939bc98
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/crates/nu-cli/src/commands.rs:37:29
  21:     0x562dc142a635 - nu::run::run_commands::h61a6206c70952e3b
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/src/run.rs:158:19
  22:     0x562dc1445bb3 - nu::main::h7b8665fb2568403e
                               at /home/disc/a.stevan/.local/share/gitstore/github.com/nushell/nushell/src/main.rs:231:9
  23:     0x562dc141c7cb - core::ops::function::FnOnce::call_once::he6b99eaca0ec581d
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
  24:     0x562dc141b0fe - std::sys_common::backtrace::__rust_begin_short_backtrace::h15d55510aa9ec694
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/sys_common/backtrace.rs:121:18
  25:     0x562dc141d301 - std::rt::lang_start::{{closure}}::h644ff56691dbb3fe
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:166:18
  26:     0x562dc41ddb8b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h8cbb48ae40ddb046
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:286:13
  27:     0x562dc41ddb8b - std::panicking::try::do_call::h92db802eb38b49b7
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
  28:     0x562dc41ddb8b - std::panicking::try::ha8949d2082cf3644
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
  29:     0x562dc41ddb8b - std::panic::catch_unwind::h5e34c1f8a5992ed9
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
  30:     0x562dc41ddb8b - std::rt::lang_start_internal::{{closure}}::hea52a0bb3f8ff16a
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:48
  31:     0x562dc41ddb8b - std::panicking::try::do_call::h5bc358faf3d68a8b
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:483:40
  32:     0x562dc41ddb8b - std::panicking::try::h675304212928379d
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:447:19
  33:     0x562dc41ddb8b - std::panic::catch_unwind::h7ce3ad349ed5c844
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panic.rs:137:14
  34:     0x562dc41ddb8b - std::rt::lang_start_internal::hcd7e45acd25ab5ab
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:148:20
  35:     0x562dc141d2da - std::rt::lang_start::h2d36e0a92261b7a9
                               at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/rt.rs:165:17
  36:     0x562dc14463fe - main
  37:     0x7f7f52829d90 - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  38:     0x7f7f52829e40 - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:392:3
  39:     0x562dc141a035 - _start
  40:                0x0 - <unknown>

on ubuntu and a very similar one on arch as well 👍

amtoine · 2023-03-30T16:54:06Z

@kubouch top comment updated 👌

amtoine · 2023-04-01T08:39:49Z

following the backtrace, in parser.rs, before the call to get_block_mut

In theory, we should only be updating captures where we have new information
the only place where this is possible would be blocks that are newly created
by our working set delta. If we ever tried to modify the permanent state, we'd
panic (again, in theory, this shouldn't be possible)

...

If we ever tried to modify the permanent state, we'd panic (again, in theory, this shouldn't be possible)

👀

This commit - 👍 does not crash anymore when running a command from `std` - 👎 loads all the commands from `std`

This actually solves the issue of loading everything from `std`!!

This commit is here to ensure the bug i had at the beginning does not happen again: - the prelude is loaded - but running a command from the prelude, or from a `use std ...` crashes the shell This commit runs the `assert` command, which is in the prelude, and needs to pass.

amtoine · 2023-04-02T09:36:12Z

standard library, here you come 😤 💪

The clip command has been introduced to the standard library in nushell/nushell/pull/8695 and has been available since nushell/nushell/pull/8627.

# Description as we now have a prelude thanks to #8627, i'd like to work on the structure of the library 😋 and i think the first step is to make it a true standalone crate 😏 this PR - moves all the library from `crates/nu-utils/standard_library/` to `crates/nu-std/` - moves the `rust` loading code from `src/run.rs` to `crates/nu-std/src/lib.rs`

amtoine mentioned this pull request Mar 26, 2023

standard library: a roadmap #8450

Open

7 tasks

fdncred added the A:std-library Defining and improving the standard library written in Nu label Mar 26, 2023

amtoine added 4 commits March 29, 2023 18:45

add a function to get the content of the standard library

525f66a

register std and import assert as a first draft

85c1102

merge the std module and loaded commands into the engine_state

fc3a42d

prepare for the help commands, once nushell#8505 is merged

72f52fb

amtoine force-pushed the feature/stdlib/add-a-prelude-automatic-loading branch from 0d05fd4 to 72f52fb Compare March 29, 2023 16:51

amtoine mentioned this pull request Mar 29, 2023

standard library: implement help commands with $nu #8505

Merged

6 tasks

kubouch reviewed Mar 29, 2023

View reviewed changes

amtoine added 6 commits March 30, 2023 18:39

give the module comments to have a nice help std

61436bc

accumulate loading errors and show a single nice one at the end

fccbcb0

add the block of std to the working set

d0101dd

report an error when the parsing fails

a503868

refactor the prelude in a function

3d01588

load the standard library with scripts and commands

6f239f9

amtoine added 4 commits April 2, 2023 10:36

parse the whole standard library

b7c3ef6

This commit - 👍 does not crash anymore when running a command from `std` - 👎 loads all the commands from `std`

do not add the block as it does not change anything

fd99c0e

parse the module as "scoped"

049f8bc

This actually solves the issue of loading everything from `std`!!

add some tests to check the prelude works

7d20d70

amtoine added 4 commits April 2, 2023 11:17

use the true URL to the issue tracker

ba065ee

move the get_standard_library tool to the top

992ba5b

refactor the prelude loading in function

8753bbe

amtoine marked this pull request as ready for review April 2, 2023 09:35

kubouch merged commit 7bac0b4 into nushell:main Apr 5, 2023

amtoine deleted the feature/stdlib/add-a-prelude-automatic-loading branch April 5, 2023 20:46

amtoine mentioned this pull request Apr 6, 2023

stdlib: make the library a standalone crate #8770

Merged

amtoine added a commit to goatfiles/nu_scripts that referenced this pull request Apr 6, 2023

use the new clip from the standard library

1fa8226

The clip command has been introduced to the standard library in nushell/nushell/pull/8695 and has been available since nushell/nushell/pull/8627.

amtoine mentioned this pull request Apr 8, 2023

stdlib: the tests do not run using the prelude #8809

Closed

Conversation

amtoine commented Mar 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

User-Facing Changes

normal behaviour

when the prelude can not be loaded

Tests + Formatting

After Submitting

Uh oh!

amtoine commented Mar 29, 2023

Uh oh!

amtoine commented Mar 29, 2023

reproduce the 👍 and the 🐛

❗ TL;DR: almost there 😌

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kubouch commented Mar 29, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kubouch commented Mar 29, 2023

Uh oh!

amtoine commented Mar 30, 2023

Uh oh!

amtoine commented Mar 30, 2023

Uh oh!

amtoine commented Mar 30, 2023

Uh oh!

amtoine commented Apr 1, 2023

Uh oh!

amtoine commented Apr 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

amtoine commented Mar 26, 2023 •

edited

Loading