Allow filesystem commands to access files with glob metachars in name#10694
Allow filesystem commands to access files with glob metachars in name#10694fdncred merged 10 commits intonushell:mainfrom
Conversation
remove deprecated MatchOptions::new() from doc comments.
command args that might be path or glob; fix 10571 by checking first for existing file before globbing.
|
I just ran I think the problem is that you're using an illegal character in Windows |
|
Hoorah, tests all pass! |
|
I added some thoughts and context here #10364 (comment) |
|
Well, I think the PR does close those issues, but it's up to the openers there to say. I left a comment in each, inviting the opener to take a look |
ooh right, i missed them, my bad 🙏 |
@fdncred But, even it were perfectly aligned, |
|
@bobhy You need to look at the ci.yml. For clippy, I currently don't like how this is separated because it makes it hard to diagnose when there are problems, but if you look here nushell/.github/workflows/ci.yml Line 12 in 4e5a1ce and then here nushell/.github/workflows/ci.yml Line 51 in 4e5a1ce For tests here nushell/.github/workflows/ci.yml Line 55 in 4e5a1ce Plugins have their own section. As it relates to Windows, you can definitely use |
|
Thanks @bobhy. Anxious to see how people like this. |
|
Yah, me too. We'll never hear from the folks for whom it just does the "right" thing. I want to hear from the folks who say, "I typed |
|
The one thing that has always concerned me about this PR is |
|
OK, but it's small potatoes. I'm doing 1 |
…nushell#10694) (squashed version of nushell#10557, clean commit history and review thread) Fixes nushell#10571, also potentially: nushell#10364, nushell#10211, nushell#9558, nushell#9310, # Description Changes processing of arguments to filesystem commands that are source paths or globs. Applies to `cp, cp-old, mv, rm, du` but not `ls` (because it uses a different globbing interface) or `glob` (because it uses a different globbing library). The core of the change is to lookup the argument first as a file and only glob if it is not. That way, a path containing glob metacharacters can be referenced without glob quoting, though it will have to be single quoted to avoid nushell parsing. Before: A file path that looks like a glob is not matched by the glob specified as a (source) argument and takes some thinking about to access. You might say the glob pattern shadows a file with the same spelling. ``` > ls a* ╭───┬────────┬──────┬──────┬────────────────╮ │ # │ name │ type │ size │ modified │ ├───┼────────┼──────┼──────┼────────────────┤ │ 0 │ a[bc]d │ file │ 0 B │ 34 seconds ago │ │ 1 │ abd │ file │ 0 B │ now │ │ 2 │ acd │ file │ 0 B │ now │ ╰───┴────────┴──────┴──────┴────────────────╯ > cp --verbose 'a[bc]d' dest copied /home/bobhy/src/rust/work/r4/abd to /home/bobhy/src/rust/work/r4/dest/abd copied /home/bobhy/src/rust/work/r4/acd to /home/bobhy/src/rust/work/r4/dest/acd > ## Note -- a[bc]d *not* copied, and seemingly hard to access. > cp --verbose 'a\[bc\]d' dest Error: × No matches found ╭─[entry nushell#33:1:1] 1 │ cp --verbose 'a\[bc\]d' dest · ─────┬──── · ╰── no matches found ╰──── > #.. but is accessible with enough glob quoting. > cp --verbose 'a[[]bc[]]d' dest copied /home/bobhy/src/rust/work/r4/a[bc]d to /home/bobhy/src/rust/work/r4/dest/a[bc]d ``` Before_2: if file has glob metachars but isn't a valid pattern, user gets a confusing error: ``` > touch 'a[b' > cp 'a[b' dest Error: × Pattern syntax error near position 30: invalid range pattern ╭─[entry nushell#13:1:1] 1 │ cp 'a[b' dest · ──┬── · ╰── invalid pattern ╰──── ``` After: Args to cp, mv, etc. are tried first as literal files, and only as globs if not found to be files. ``` > cp --verbose 'a[bc]d' dest copied /home/bobhy/src/rust/work/r4/a[bc]d to /home/bobhy/src/rust/work/r4/dest/a[bc]d > cp --verbose '[a][bc]d' dest copied /home/bobhy/src/rust/work/r4/abd to /home/bobhy/src/rust/work/r4/dest/abd copied /home/bobhy/src/rust/work/r4/acd to /home/bobhy/src/rust/work/r4/dest/acd ``` After_2: file with glob metachars but invalid pattern just works. (though Windows does not allow file name to contain `*`.). ``` > cp --verbose 'a[b' dest copied /home/bobhy/src/rust/work/r4/a[b to /home/bobhy/src/rust/work/r4/dest/a[b ``` So, with this fix, a file shadows a glob pattern with the same spelling. If you have such a file and really want to use the glob pattern, you will have to glob quote some of the characters in the pattern. I think that's less confusing to the user: if ls shows a file with a weird name, s/he'll still be able to copy, rename or delete it. # User-Facing Changes Could break some existing scripts. If user happened to have a file with a globbish name but was using a glob pattern with the same spelling, the new version will process the file and not expand the glob. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
…nushell#10694) (squashed version of nushell#10557, clean commit history and review thread) Fixes nushell#10571, also potentially: nushell#10364, nushell#10211, nushell#9558, nushell#9310, # Description Changes processing of arguments to filesystem commands that are source paths or globs. Applies to `cp, cp-old, mv, rm, du` but not `ls` (because it uses a different globbing interface) or `glob` (because it uses a different globbing library). The core of the change is to lookup the argument first as a file and only glob if it is not. That way, a path containing glob metacharacters can be referenced without glob quoting, though it will have to be single quoted to avoid nushell parsing. Before: A file path that looks like a glob is not matched by the glob specified as a (source) argument and takes some thinking about to access. You might say the glob pattern shadows a file with the same spelling. ``` > ls a* ╭───┬────────┬──────┬──────┬────────────────╮ │ # │ name │ type │ size │ modified │ ├───┼────────┼──────┼──────┼────────────────┤ │ 0 │ a[bc]d │ file │ 0 B │ 34 seconds ago │ │ 1 │ abd │ file │ 0 B │ now │ │ 2 │ acd │ file │ 0 B │ now │ ╰───┴────────┴──────┴──────┴────────────────╯ > cp --verbose 'a[bc]d' dest copied /home/bobhy/src/rust/work/r4/abd to /home/bobhy/src/rust/work/r4/dest/abd copied /home/bobhy/src/rust/work/r4/acd to /home/bobhy/src/rust/work/r4/dest/acd > ## Note -- a[bc]d *not* copied, and seemingly hard to access. > cp --verbose 'a\[bc\]d' dest Error: × No matches found ╭─[entry nushell#33:1:1] 1 │ cp --verbose 'a\[bc\]d' dest · ─────┬──── · ╰── no matches found ╰──── > #.. but is accessible with enough glob quoting. > cp --verbose 'a[[]bc[]]d' dest copied /home/bobhy/src/rust/work/r4/a[bc]d to /home/bobhy/src/rust/work/r4/dest/a[bc]d ``` Before_2: if file has glob metachars but isn't a valid pattern, user gets a confusing error: ``` > touch 'a[b' > cp 'a[b' dest Error: × Pattern syntax error near position 30: invalid range pattern ╭─[entry nushell#13:1:1] 1 │ cp 'a[b' dest · ──┬── · ╰── invalid pattern ╰──── ``` After: Args to cp, mv, etc. are tried first as literal files, and only as globs if not found to be files. ``` > cp --verbose 'a[bc]d' dest copied /home/bobhy/src/rust/work/r4/a[bc]d to /home/bobhy/src/rust/work/r4/dest/a[bc]d > cp --verbose '[a][bc]d' dest copied /home/bobhy/src/rust/work/r4/abd to /home/bobhy/src/rust/work/r4/dest/abd copied /home/bobhy/src/rust/work/r4/acd to /home/bobhy/src/rust/work/r4/dest/acd ``` After_2: file with glob metachars but invalid pattern just works. (though Windows does not allow file name to contain `*`.). ``` > cp --verbose 'a[b' dest copied /home/bobhy/src/rust/work/r4/a[b to /home/bobhy/src/rust/work/r4/dest/a[b ``` So, with this fix, a file shadows a glob pattern with the same spelling. If you have such a file and really want to use the glob pattern, you will have to glob quote some of the characters in the pattern. I think that's less confusing to the user: if ls shows a file with a weird name, s/he'll still be able to copy, rename or delete it. # User-Facing Changes Could break some existing scripts. If user happened to have a file with a globbish name but was using a glob pattern with the same spelling, the new version will process the file and not expand the glob. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
| } | ||
|
|
||
| // user wasn't referring to a specific thing in filesystem, try to glob it. | ||
| match glob_with_parent(&pattern.item, options, cwd) { |
There was a problem hiding this comment.
Hi, @bobhy
Sorry for a simple question, why are you using glob_with_parent rather than glob_with?
…`du` commands (#11621) # Description This pr is a follow up to [#11569](#11569 (comment)) > Revert the logic in #10694 and apply the logic in this pr to mv, cp, rv will require a larger change, I need to think how to achieve the bahavior And sorry @bobhy for reverting some of your changes. This pr is going to unify glob behavior on the given commands: * open * rm * cp-old * mv * umv * cp * du So they have the same behavior to `ls`, which is: If given parameter is quoted by single quote(`'`) or double quote(`"`), don't auto-expand the glob pattern. If not quoted, auto-expand the glob pattern. Fixes: #9558 Fixes: #10211 Fixes: #9310 Fixes: #10364 # TODO But there is one thing remains: if we give a variable to the command, it will always auto-expand the glob pattern, e.g: ```nushell let path = "a[123]b" rm $path ``` I don't think it's expected. But I also think user might want to auto-expand the glob pattern in variables. So I'll introduce a new command called `glob escape`, then if user doesn't want to auto-expand the glob pattern, he can just do this: `rm ($path | glob escape)` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Done # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> ## NOTE This pr changes the semantic of `GlobPattern`, before this pr, it will `expand path` after evaluated, this makes `nu_engine::glob_from` have no chance to glob things right if a path contains glob pattern. e.g: [#9310 ](#9310 (comment)) #10211 I think changing the semantic is fine, because it makes glob works if path contains something like '*'. It maybe a breaking change if a custom command's argument are annotated by `: glob`.
…`du` commands (nushell#11621) # Description This pr is a follow up to [nushell#11569](nushell#11569 (comment)) > Revert the logic in nushell#10694 and apply the logic in this pr to mv, cp, rv will require a larger change, I need to think how to achieve the bahavior And sorry @bobhy for reverting some of your changes. This pr is going to unify glob behavior on the given commands: * open * rm * cp-old * mv * umv * cp * du So they have the same behavior to `ls`, which is: If given parameter is quoted by single quote(`'`) or double quote(`"`), don't auto-expand the glob pattern. If not quoted, auto-expand the glob pattern. Fixes: nushell#9558 Fixes: nushell#10211 Fixes: nushell#9310 Fixes: nushell#10364 # TODO But there is one thing remains: if we give a variable to the command, it will always auto-expand the glob pattern, e.g: ```nushell let path = "a[123]b" rm $path ``` I don't think it's expected. But I also think user might want to auto-expand the glob pattern in variables. So I'll introduce a new command called `glob escape`, then if user doesn't want to auto-expand the glob pattern, he can just do this: `rm ($path | glob escape)` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Done # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> ## NOTE This pr changes the semantic of `GlobPattern`, before this pr, it will `expand path` after evaluated, this makes `nu_engine::glob_from` have no chance to glob things right if a path contains glob pattern. e.g: [nushell#9310 ](nushell#9310 (comment)) nushell#10211 I think changing the semantic is fine, because it makes glob works if path contains something like '*'. It maybe a breaking change if a custom command's argument are annotated by `: glob`.
(squashed version of #10557, clean commit history and review thread)
Fixes #10571, also potentially: #10364, #10211, #9558, #9310,
Description
Changes processing of arguments to filesystem commands that are source paths or globs.
Applies to
cp, cp-old, mv, rm, dubut notls(because it uses a different globbing interface) orglob(because it uses a different globbing library).The core of the change is to lookup the argument first as a file and only glob if it is not. That way,
a path containing glob metacharacters can be referenced without glob quoting, though it will have to be single quoted to avoid nushell parsing.
Before: A file path that looks like a glob is not matched by the glob specified as a (source) argument and takes some thinking about to access. You might say the glob pattern shadows a file with the same spelling.
Before_2: if file has glob metachars but isn't a valid pattern, user gets a confusing error:
After: Args to cp, mv, etc. are tried first as literal files, and only as globs if not found to be files.
After_2: file with glob metachars but invalid pattern just works. (though Windows does not allow file name to contain
*.).So, with this fix, a file shadows a glob pattern with the same spelling. If you have such a file and really want to use the glob pattern, you will have to glob quote some of the characters in the pattern. I think that's less confusing to the user: if ls shows a file with a weird name, s/he'll still be able to copy, rename or delete it.
User-Facing Changes
Could break some existing scripts. If user happened to have a file with a globbish name but was using a glob pattern with the same spelling, the new version will process the file and not expand the glob.
Tests + Formatting
After Submitting