Skip to content

Load default_env.nu/default_config.nu before user env.nu/config.nu#14249

Merged
fdncred merged 27 commits intonushell:mainfrom
NotTheDr01ds:always-load-defaults
Nov 20, 2024
Merged

Load default_env.nu/default_config.nu before user env.nu/config.nu#14249
fdncred merged 27 commits intonushell:mainfrom
NotTheDr01ds:always-load-defaults

Conversation

@NotTheDr01ds
Copy link
Copy Markdown
Contributor

@NotTheDr01ds NotTheDr01ds commented Nov 4, 2024

Release-Notes Short Description

  • Nushell now always loads its internal default_env.nu before the user env.nu is loaded, then loads the internal default_config.nu before the user's config.nu is loaded. This allows for a simpler user-configuration experience. Details are in this blog entry along with an updated Configuration Chapter

Description

Implements the main ideas in #13671 and a few more:

  • Users can now specify only the environment and config options they want to override in their env.nu and config.nuand yet still have access to all of the defaults:
    • default_env.nu (internally defined) will be loaded whenever (and before) the user's env.nu is loaded.
    • default_config.nu (internally defined) will be loaded whenever (and before) the user's config.nu is loaded.
  • No more 900+ line config out-of-the-box.
  • Faster startup (again): ~40-45% improvement in launch time with a default configuration.
  • New keys that are added to the defaults in the future will automatically be available to all users after updating Nushell. No need to regenerate config to get the new defaults.
  • It is now possible to have different internal defaults (which will be used with -c and scripts) vs. REPL defaults. This would have solved many of the user complaints about the display_errors implementation.
  • A basic "scaffold" config.nu and env.nu are created on first launch (if the config directory isn't present).
  • Improved "out-of-the-box" experience (OOBE) - No longer asks to create the files; the minimal scaffolding will be automatically created. If deleted, they will not be regenerated. This provides a better "out-of-the-box" experience for the user as they no longer have to make this decision (without much info on the pros or cons) when first launching.
  • (New: 2024-11-07) Runs the env_conversions process after the default_env.nu is loaded so that users can treat Path/PATH as lists in their own config.
  • (New: 2024-11-08) Given the changes in Refactor config updates #13802, default_config.nu will be a minimal file to minimize load-times. This shaves another (on my system) ~3ms off the base launch time.
    • Related: Keybindings, menus, and hooks that are already internal defaults are no longer duplicated in $env.config. The documentation will be updated to cover these scenarios.
  • (New: 2024-11-08) Move existing "full" default_config.nu to sample_config.nu for short-term "documentation" purposes.
  • (New: 2024-11-18) Move the dark-theme and light-theme to Standard Library and demonstrate their use - Also improves startup times, but we're reaching the limit of optimization.
  • (New: 2024-11-18) Extensively documented/commented sample_env.nu and sample_config.nu. These can be displayed in-shell using (for example) config nu --sample | nu-highlight | less -R. Note: Much of this will eventually be moved to or (some) duplicated in the Doc. But for now, this some nice in-shell doc that replaces the older "commented/documented default".
  • (New: 2024-11-20) Runs the ENV_CONVERSIONS process (1) after the default_env.nu (allows PATH to be used as a list in user's env.nu) and (2) before default_config.nu is loaded (allows user's ENV_CONVERSIONS from their env.nu to be used in their config.nu).
  • (New: 2024-11-20) The default ENV_CONVERSIONS is now an empty record. The internal Rust code handles PATH (and variants) conversions regardless of the ENV_CONVERSIONS variable. This shaves a very small amount of time off the startup. Reset - Looks like there might be a bug in nu-enginer::env::ensure_path() on Windows that would need to be fixed in order for this to work.

User-Facing Changes

By default, you shouldn't see much, if any, change when running this with your existing configuration.

To see the greatest benefit from these changes, you'll probably want to start with a "fresh" config. This can be easily tested using something like:

let temp_home = (mktemp -d)
$env.XDG_CONFIG_HOME = $temp_home
$env.XDG_DATA_HOME = $temp_home
./target/release/nu

You should see a message where the (mostly empty) env.nu and config.nu are created on first start. Defaults should be the same (or similar to) those before the PR. Please let me know if you notice any differences.


Users should now specify configuration in terms of overrides of each setting. For instance, rather than modifying history settings in the monolithic config.nu, the following is recommended in an updated config.nu:

$env.config.history = {
  file_format: sqlite,
  sync_on_enter: true
  isolation: true
  max_size: 1_000_000
}

or even just:

$env.config.history.file_format = sqlite
$env.config.history.isolation: true
$env.config.history.max_size = 1_000_000

Note: It seems many users are already appending a source my_config.nu (or similar pattern) to the end of the existing config.nu to make updates easier. In this case, they will likely want to remove all of the previous defaults and just move their my_config.nu to config.nu.

Note: It should be unlikely that there are any breaking changes here, but there's a slim chance that some code, somewhere, expects an absence of certain config values. Otherwise, all config values are available before and after this change.

Tests + Formatting

  • 🟢 toolkit fmt
  • 🟢 toolkit clippy
  • 🟢 toolkit test
  • 🟢 toolkit test stdlib

After Submitting

Configuration Chapter (and related) of the doc is currently WIP and will be finished in time for 0.101 release.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Nov 4, 2024

@fdncred For your gist on this topic, I believe we'll find that default_config.nu and default_env.nu are loaded in every case where your gist shows config.nu and env.nu as loading. With, of course, the exception of --no-defaults.

For example, if --no-config-file or -c are specified, then none of the config files loaded, including default_env.nu, env.nu, default_config.nu, and config.nu.

@NotTheDr01ds NotTheDr01ds changed the title Always load default config contents Always load default env/config values Nov 4, 2024
@sholderbach sholderbach added the A:configuration Issue related to nu's configuration label Nov 4, 2024
@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Nov 4, 2024

Edit: This was implement:

I'm thinking (but not totally sure yet) that we might could also move the "path conversion" code so that it runs after the default_env.nu. If we could, this would allow the $env.PATH to be treated as a Nushell list inside env.nu.

@IanManske
Copy link
Copy Markdown
Member

Thanks for the PR! Just tried this out, but it look likes $env.config is undefined on startup.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Nov 5, 2024

@IanManske Thanks for testing! Can you give me some more detail on your startup?

  • I'm assuming you are using a fresh install or XDG_CONFIG_HOME since otherwise your $env.config should still be getting (at least) populated from the existing config, right?
  • If there's no initial config/env, what is getting generated when you startup with this PR?
  • Does view files | find config and view files | find env show the defaults loading?
  • Platform?
  • Anything else you can tell me to help me reproduce?

Thanks!

@IanManske
Copy link
Copy Markdown
Member

Sure thing, I'm on 938cd2c and ran:

  • cargo b
  • ./target/debug/nu -n
  • $env.config

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Nov 6, 2024

@IanManske Ah - Thanks! That's working as expected then. When you run with -n we don't load any config/env files, and that includes the defaults. You can confirm this with the view files command I mentioned above.

If you want to test in a clean environment:

$env.XDG_CONFIG_HOME = (mktemp -d)
./target/debug/nu

You should see something like:

Environment config file created at: /tmp/tmp.n4eZIEkAuG/nushell/env.nu
Config file created at: /tmp/tmp.n4eZIEkAuG/nushell/config.nu

... followed by the banner. And $env.confg should be fully populated there.

After exiting that shell, you can clean up with:

rm -r $env.XDG_CONFIG_HOME
hide-env XDG_CONFIG_HOME

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Nov 6, 2024

@fdncred For your gist on this topic, I believe we'll find that default_config.nu and default_env.nu are loaded in every case where your gist shows config.nu and env.nu as loading.

@NotTheDr01ds it would be good if someone would take my gist and turn it into some documentation with these changes you're talking about.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

@fdncred Sure thing - I'll get that in when I do that Config chapter update/rewrite for these changes.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Nov 6, 2024

In today's meeting, we discussed several options that might allow users to disable the loading of defaults:

  1. The flag in place at the moment. Disadvantage is that (a) it's another flag, and (b) it would have to be added every time.
  2. A "magic" file named something like .no-defaults that, if present in the config directory, would prevent defaults from loading. This was seen as an improvement over the flag, but general consensus was that it is a bit "hokey" (my wording).
  3. Instead of auto-loading, move the defaults to a config subcommand that the user would call in their own env.nu/config.nu. This might work, but I'm not sure we could still allow the environment conversions to work in that case. I'd have to play with it to see. This would also be a solution where the (we believe) minority of users would "do nothing" to not load the defaults, but the (we believe) vast majority would have to add a command to both env.nu and config.nu.
  4. Ultimately we decided that it's probably best to see if this is really going to be a problem before we decide on or implement a solution. If most users just load the defaults and then shift to an "override" config, then we might not need a solution at all.

So I'm going to back out the flag.

We also decided we will hold this until after 0.100.0 is released. This will give us more time for it to bake in nightly's/etc.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

Also, as requested during today's meeting, here's a link to the gist.

@NotTheDr01ds NotTheDr01ds marked this pull request as draft November 7, 2024 01:09
@stormasm
Copy link
Copy Markdown
Contributor

stormasm commented Nov 7, 2024

@NotTheDr01ds Thank you kindly for adding a summary of what we discussed in the meeting today 😄

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

One thing that I honestly didn't notice until after the call yesterday is just how substantially @IanManske's #13802 changed the way config works. I usually just gloss over PR's with "refactor" in the title ;-).

That's a great change, and does a lot of what this PR would have done if #13802 wasn't in place.

But the combination of both of these gives us even more opportunities, I think.

I'm thinking of now changing default_config.nu to (for starters) be:

$env.config = {}

Because of Ian's work, this will now populate all values from impl Default for Config into the record.

And ... Sweet! This shaves another (average) ~3ms (on my system) off a default/base launch time compared to 0.99.1.

There might be a few values we want or need to populate back in, so that the "internal defaults" and default_config.nu values totally sync up, but for the most part the Rust-internal-defaults will give us (another) decent startup-performance boost.

@IanManske
Copy link
Copy Markdown
Member

IanManske commented Nov 8, 2024

Some of the config settings will probably be hard to set through the Rust code like the hooks and theme, so I agree we probably still want something (or most things?) in the default_config.nu. Plus this would be accessible through config nu --default as a temporary form of documentation.

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Nov 8, 2024

Related / maybe Unrelated

Just posting what I posted on Discord
taking a cue from ghostty, it may be nice to have a functionality that does this :

  1. ghostty +show-config <-- shows the current configuration settings in a valid ghostty config format. 278 lines
  2. ghostty +show-config --default <-- shows every single option with its default (without a config file) value 442 lines
  3. ghostty +show-config --default --docs <-- shows every single option with its default (without a config file) value + documentation that explains every options use case. 1,960 lines
  4. ghostty +show-config --changes-only <-- only show the options that have been changed from the default, no effect if --default is specified

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

Plus this would be accessible through config nu --default as a temporary form of documentation.

At least temporarily, I'm planning on moving the current (fully-loaded) version of default_config.nu to sample_config.nu and having it be accessible through the config command for this purpose. Whether as config nu --default or config nu --sample or whatever. I'm not sure that's the best "documentation", but it will service temporarily.

Long-term, I'd like to implement something like @fdncred pointed out with the Ghostty stuff at some point. In our case, though, it would be something like:

# Flattened tables
help config-option --list/-l --current
help config-option --list-l --default
help config-option --list-l --overrides

# Perhaps, to avoid too deeply nested a structure above:
help keybindings ...
help hooks ...

# Show help on option
help config-option <option>

I wonder if we could add the documentation on each setting directly to the configuration as we do with commands?

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Nov 9, 2024

I kind of feel like there needs to be a readme.md in the default_files folder that describes the config files. we have 3 config files, config, env, login. some of those have default_ files, some have scaffold_ files, some have sample_ files but it's not the same for all 3 groups so it's a bit confusing.

default_

has config and env but no login

scaffold_

has config and env but no login

sample_

has config and login but no env

even after reading the code i'm confused about all these files, what their purpose is, and when they're loaded. LOL.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

@fdncred Fair enough - It's (a) confusing, and (b) probably in flux regardless. I'll add a README there, but long-term, I think both the sample_ files go away.

I'm not even sure what the purpose of sample_login.nu is - It's not used anywhere in the code, so I'm going to move the info there to the Configuration chapter of the Book and then remove it.

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Nov 9, 2024

sample_login.nu was there to (1) let people know that there could be a login.nu file and (2) what the syntax needed to look like and for documentation purposes.

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

Right, but it wasn't actually used - I would have thought it would be written to the config directory, but it's not. I don't think most users were going to find it at crates/nu-utils/src/sample_files/sample_login.nu ;-)

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Nov 11, 2024

I'm trying to test this out on Windows my setting the XDG_CONFIG_HOME var but I'm having problems.

If I have this I get an error.

$env.config.table = {
  show_empty: false
  trim: {
    truncating_suffix: ""
  }
  header_on_separator: true
  footer_inheritance: true
}
Error: nu::shell::invalid_config

  × Encountered 1 error(s) when updating config

Error: nu::shell::missing_required_column

  × $env.config.table.trim requires a 'methodology' column
    ╭─[C:\Users\fdncred\.config\nushell\config.nu:96:9]
 95 │       show_empty: false
 96 │ ╭─▶   trim: {
 97 │ │       truncating_suffix: "…"
 98 │ ├─▶   }
    · ╰──── has no 'methodology' column
 99 │       header_on_separator: true
    ╰────

If I replace it with my entire $env.config.table section it gets passed it and seems to work. I'm wondering if:
A. I've done something wrong with my config.
B. There are some dependencies in some of the config sections where particular sections are required like methodology.

Here's my config & env in case someone can spot something wrong.
env.nu.txt
config.nu.txt

@fdncred fdncred merged commit 4ed25b6 into nushell:main Nov 20, 2024
@github-actions github-actions bot added this to the v0.101.0 milestone Nov 20, 2024
@rgwood
Copy link
Copy Markdown
Contributor

rgwood commented Nov 23, 2024

This looks like a big improvement+simplification. Thank you!

@NotTheDr01ds NotTheDr01ds changed the title Always load default env/config values Load default_env.nu/default_config.nu before user env.nu/config.nu Nov 24, 2024
WindSoilder pushed a commit that referenced this pull request Nov 27, 2024
# Description

As a bit of a follow-on to #13802 and #14249, this (pretty much a
"one-line" change) really does *always* populate the `$env.config`
record with the `nu-protocol::config` defaults during startup. This
means that an `$env.config` record is value (with defaults) even during:

* `nu -n` to suppress loading of config files
* `nu -c <commandstring>`
* `nu <script>`

# User-Facing Changes

There should be no case in which there isn't a valid `$env.config`.

* Before:

  ```nushell
  nu -c "$env.config"
  # -> Error
  ```

* After:

  ```nushell
  nu -c "$env.config"
  # -> Default $env.config record
  ```

Startup time impact is negligible (17.072µs from `perf!` on my system) -
Seems well worth it.

# Tests + Formatting

Added tests for several `-n -c` cases.

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting

Config chapter update still in progress.
fdncred pushed a commit that referenced this pull request Nov 28, 2024
# Description

Due to #14249 loading `default_env.nu` before the user's `env.nu`,
variables that were defined there were overriding:

* Inherited values
* Some values that were set in the Rust code, such as the `NU_LIB_PATH`
when set using `--include-path`.

This change checks to see if a variable already exists, uses its value
if so, and sets the default value otherwise.

Note: `ENV_CONVERSIONS` is still "forced" to a default value regardless,
as it needs to run reliably. There's probably not much reason to inherit
it, but I'm open to the idea if there's a use-case.

# User-Facing Changes

* Before: Variables that were set in `default_env.nu` always overrode
those that were inherited from the parent process or set internally
* After: Inherited and internal environment variables will take
priority.

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting

Will try to find a good place to mention this behavior in the Config
chapter updates
fdncred pushed a commit that referenced this pull request Dec 10, 2024
# Description

#14249 loaded `convert_env_values()` several times to force more updates
to `ENV_CONVERSION`. This allows the user to treat variables as
structured data inside `config.nu` (and others).

Unfortunately, `convert_env_values()` did not originally anticipate
being called more than once, so it would attempt to re-convert values
that had already been converted. This usually leads to an error in the
conversion closure.

With this PR, values are only converted with `from_string` if they are
still strings; otherwise they are skipped and their existing value is
used.

# User-Facing Changes

No user-facing change when compared to 0.100, since closures written for
0.100's `ENV_CONVERSION` now work again without errors.

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
- 
# After Submitting

Will remove the "workaround" from the Config doc preview.
@@ -0,0 +1,792 @@
# Nushell Config File
#
# version = "0.99.2"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong version?

Copy link
Copy Markdown
Contributor

@fdncred fdncred Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PerchunPak Good catch. @NotTheDr01ds I'm also thinking all these config/env/scaffold/sample files should have a version at the top. Also, I'm getting an error in one. It should be = not :.
image

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are many syntax errors in the nightly build, not only this one



and many more

i will probably create a pr fixing all those

fdncred pushed a commit that referenced this pull request Dec 16, 2024
…nt prompt vars (#14579)

# Description

With `NU_LIB_DIRS`, `NU_PLUGIN_DIRS`, and `ENV_CONVERSIONS` now moved
out of `default_env.nu`, we're down to just a few left. This moves all
non-closure `PROMPT` variables out as well (and into Rust `main()`. It
also:

* Implements #14565 and sets the default
`TRANSIENT_PROMPT_COMMAND_RIGHT` and `TRANSIENT_MULTILINE_INDICATOR` to
an empty string so that they are removed for easier copying from the
terminal.
* Reverses portions of #14249 where I was overzealous in some of the
variables that were imported
* Fixes #12096 
* Will be the final fix in place, I believe, to close #13670

# User-Facing Changes

Transient prompt will now remove the right-prompt and
multiline-indicator once a commandline has been entered.

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
- 
# After Submitting

Release notes addition
@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

NotTheDr01ds commented Dec 16, 2024

@PerchunPak Yes, the conversion from the old default_config.nu with a monolithic record to the new "sample" with individual key-value settings was a pretty manual, so I missed a few that needed to be changed like this. Thanks for catching these!

As for the version numbers, I assumed these were added automatically during the build process, but I didn't dive in to be sure. I'm fairly certain no one goes in and manually updates them each release ;-)

@fdncred I'm not sure the sample_config.nu should have a version - We've tried to make sure that doc isn't versioned in most cases, right?

@NotTheDr01ds
Copy link
Copy Markdown
Contributor Author

Actually, looking through these, none of them should be "versioned" any longer. That's a relic of having breaking changes in default configs, which #14249 does away with.

@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Dec 17, 2024

@NotTheDr01ds I think we should version all of them to demonstrate which version they're released with and compatible with. For instance, if you rename something like $env.config.<any_key_mentioned> there could be mistakes that lead to confusion for people.

fdncred pushed a commit that referenced this pull request Dec 17, 2024
For some reason, it had multiple syntax errors and other issues, like
undefined options. Would be great to add a test for sourcing all example
configs, but I don't know rust

See also
#14249 (comment)

<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

CC @NotTheDr01ds
fdncred pushed a commit that referenced this pull request Jan 2, 2025
# Description

Takes advantage of #14591 to remove the now-necessary calls to
`convert_env_values()` that I added in #14249. The function is now just
called once to convert `PATH`.

Also removed the Windows-build-time checks for `ensure_path`, since
previous case-insensitivity fixes make this unnecessary as well.

# User-Facing Changes

None - #14591 now handles conversion 'on-demand'.

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting

N/A
NotTheDr01ds pushed a commit that referenced this pull request Jan 5, 2025
In #14249, `config reset` wasn't updated to use the scaffold config files, so running `config reset` would accidentally reset the user's config to the internal defaults. This PR updates it to use the
scaffold files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A:configuration Issue related to nu's configuration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants