Skip to content

new command url parse (#6854) and url subcommands tests#7124

Merged
rgwood merged 1 commit intonushell:mainfrom
raccmonteiro:command-url-parse
Nov 19, 2022
Merged

new command url parse (#6854) and url subcommands tests#7124
rgwood merged 1 commit intonushell:mainfrom
raccmonteiro:command-url-parse

Conversation

@raccmonteiro
Copy link
Copy Markdown
Contributor

@raccmonteiro raccmonteiro commented Nov 13, 2022

Description

New command url parse, and added some tests to commands url host, url scheme, url path, url query.
image

image

Fixes #6854

Tests + Formatting

Commands with error:

"" | url parse
"aaaa" | url parse

Valid commands:

"http://www.abc.com/11/22?param1=ab&param2=zz" | url parse
"http://www.abc.com:8081/11/22?param1=ab&param2=zz" | url parse

User-Facing Changes

New command url parse.

}

fn signature(&self) -> Signature {
Signature::build("url parse")
Copy link
Copy Markdown
Contributor

@dandavison dandavison Nov 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll need to declare a vec of (input_type, output_type) pairs here -- lots of examples in the other command files. You'll see the test fail for your example until you declare these:

thread 'network::url::parse::test::test_examples' panicked at 'The example `echo 'http://www.example.com/foo/bar?p=section&s=&f[name]=vldc' | url parse` demonstrates a transformation of type String -> Record([("scheme", String), ("host", String), ("path", String), ("query", String), ("params", Record([("p", String), ("s", String), ("f[name]", String)]))]). However, this does not match the declared signature: []. (Did you forget to declare the input and output types for the command?) For this command, `vectorizes_over_list` is false and `operates_on_cell_paths()` is true.', crates/nu-command/src/example_test.rs:178:21

}

fn search_terms(&self) -> Vec<&str> {
vec!["url", "parse"]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that we don't add any words from the command name itself to the search terms: just any related terms that you believe users might enter when looking for this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm thanks for the explanation, so maybe "scheme", "hostname", "port", "path", and "parameters" ? That it's what the command returns

fn examples(&self) -> Vec<Example> {
vec![Example {
description: "Parses a url",
example: "echo 'http://www.example.com/foo/bar?p=section&s=&f[name]=vldc' | url parse",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although several commands still do it in their examples, echo isn't necessary any longer and for cases like this IMO we should omit the echo: a data structure can just be supplied as input to a pipeline.

span: head,
});
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about writing this something like the following?

vals.push(Value::String {
    val: url.port().map(|p| p.to_string()).unwrap_or("".into()),
    span: head,
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks that's much better :) I am new to Rust, I still don't know the best way to handle some things.
I'll change to your code in next commit.

for (k, v) in result {
param_cols.push(k);
param_vals.push(Value::String { val: v, span: head })
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about writing this something like this?

let (param_cols, param_vals) = result
    .into_iter()
    .map(|(k, v)| (k, Value::String { val: v, span: head }))
    .unzip();

}
Err(_e) => {
return Err(ShellError::UnsupportedInput(
"Incomplete or incorrect URL. Expected a full Url, e.g., https://www.example.com"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent capitalization in error message. (It's nice to have our error messages perfect!)

@dandavison
Copy link
Copy Markdown
Contributor

There are some clippy failures to be fixed before the CI will run the tests.

@dandavison
Copy link
Copy Markdown
Contributor

dandavison commented Nov 13, 2022

Thanks so much for doing this -- it's a really nicely executed PR with great tests and documentation.

There are some related design issues, which I'm just going to list, but sorting all this out is probably a separate issue.

  • we shouldn't have this and from url at the same time
  • '1:2' | parse '{x}:{y}' returns a table (but maybe it should not) whereas this returns a record.
  • parse takes list input whereas this does not: [ '1:2' '3:4' ] | parse '{x}:{y}' (it makes sense to return a table there)

@@ -0,0 +1,28 @@
use nu_test_support::{nu, pipeline};
Copy link
Copy Markdown
Contributor

@dandavison dandavison Nov 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should keep url path and url query and url scheme? My immediate reaction is that they should not exist if we have url parse since one should just do e.g. ($url | url parse).path.

"port".to_string(),
"path".to_string(),
"query".to_string(),
"params".to_string(),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any interest in adding the other parsed components while you're at it? I think username and password are pretty useful for people working with ssh, and fragment is common... may as well take them all?

Value::String {
val: url
.port()
.map(|p| p.to_string())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should leave this as an integer seeing as Rust is providing it as an int.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in that case, what should be the output when there's no port on url ? An empty string? I could do something like this for port value:

match url.port() {
    Some(p) => Value::Int {
        val: p as i64,
        span: head,
    },
    None => Value::String {
        val: "".into(),
        span: head,
    }
},

But this way, the url parse output type for port will be string or int depending on input.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh you're right, sorry, I think what I said doesn't make sense until/unless Nushell gains an Option type in the future. As you have it is the way for now.

@dandavison
Copy link
Copy Markdown
Contributor

Would you be able to delete all the commands that this supersedes please? url host|path|query|scheme etc.

Then this is looking great and ready to merge (just a couple of review comments)!

@raccmonteiro raccmonteiro force-pushed the command-url-parse branch 2 times, most recently from 27f3d68 to 71b2c57 Compare November 15, 2022 15:05
@dandavison
Copy link
Copy Markdown
Contributor

This looks beautiful to me. Very clean, and more functionality than before. And well tested. LGTM!

@sholderbach sholderbach added the notes:breaking-changes This PR implies a change affecting users and has to be noted in the release notes label Nov 15, 2022
Comment on lines +15 to +25
"╭──────────┬───────────────────╮\
│ scheme │ https │\
│ username │ │\
│ password │ │\
│ host │ www.abc.com │\
│ port │ │\
│ path │ / │\
│ query │ │\
│ fragment │ │\
│ params │ {record 0 fields} │\
╰──────────┴───────────────────╯"
Copy link
Copy Markdown
Contributor

@rgwood rgwood Nov 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we generally want to avoid testing the formatted output from table like this. Tests like this are fragile and will break whenever we tweak Nu's display logic.

Alternative approaches:

  1. Figure out a way to test equality against a Value::Record (instead of text output) in Rust
  2. Do the equality check against a Nu record in Nu code. Something like:
let actual = nu!(
    cwd: ".", pipeline(
        r#"
            ("https://www.abc.com" | url parse) == {scheme: 'https', username: '', ...}
        "#
));
assert_eq!(actual.out, "true");

(there's probably a better way than that, it's just the first thing that came to mind)

Copy link
Copy Markdown
Contributor Author

@raccmonteiro raccmonteiro Nov 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that makes sense, I just copied how some other tests were done. I can change the tests, just waiting for confirmation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests updated

@rgwood
Copy link
Copy Markdown
Contributor

rgwood commented Nov 16, 2022

Agreed with Dan, this looks great. Thank you for the contribution!

I just have one concern about the tests, let me know what you think.

@raccmonteiro raccmonteiro force-pushed the command-url-parse branch 2 times, most recently from d60cb8d to a186826 Compare November 17, 2022 10:38
*code refactor from PR tips & clippy fixes

*added username, password, and fragment

*commands `url host`, `url scheme`, `url query`, and `url path` removed

*tests refactoring - avoid formatted output
Copy link
Copy Markdown
Contributor

@fdncred fdncred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm really loving this cleanup, code-work, tests, and general usability improvements. This is exactly the type of PR that we need more of. Good work! I'm ready to land.

@rgwood rgwood merged commit ced5e10 into nushell:main Nov 19, 2022
@rgwood
Copy link
Copy Markdown
Contributor

rgwood commented Nov 19, 2022

Fantastic change, thanks. Because this is a breaking change (existing commands removed), could you please document it in the blog post for v0.72 when you have time? nushell/nushell.github.io#666

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

notes:breaking-changes This PR implies a change affecting users and has to be noted in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

from url doesn't seem to work well with absolute URLs

5 participants