Fix 8244 -- store timestamps with nanosecond resolution (consistently) by bobhy · Pull Request #8337 · nushell/nushell

bobhy · 2023-03-06T06:57:54Z

Description

Fix for data ambiguity noted in #8244.

Basic change is to use nanosecond resolution for unix timestamps (stored in type Int). Previously, a timestamp might have seconds, milliseconds or nanoseconds, but it turned out there were overlaps in data ranges between different resolutions, so there wasn't always a unique mapping back to date/time.

Due to higher precision, the range of dates that timestamps can map to is restricted. Unix timestamps with seconds resolution and 64 bit storage can cover all dates from the Big Bang to eternity. Timestamps with seconds resolution and 32 bit storage can only represent dates from 1901-12-13 through 2038-01-19. The nanoseconds resolution and 64 bit storage used with this fix can represent dates from 1677-09-21T00:12:44 to 2262-04-11T23:47:16, something of a compromise.

User-Facing Changes

(List of all changes that impact the user experience here. This helps us keep track of breaking changes.)

`<datetime> | into int`

Converts to nanosecond resolution

〉date now | into int
1678084730502126846

This is the number of non-leap nanoseconds after the unix epoch date: 1970-01-01T00:00:00+00:00.

Conversion fails for dates outside the supported range:

〉1492-10-12 | into int
Error: nu::shell::incorrect_value

  × Incorrect value.
   ╭─[entry #51:1:1]
 1 │ 1492-10-12 | into int
   ·              ────┬───
   ·                  ╰── DateTime out of timestamp range 1677-09-21T00:12:43 and 2262-04-11T23:47:16
   ╰────

`<int> | into datetime`

Can no longer fail or produce incorrect results for any 64-bit input:

〉0 | into datetime 
Thu, 01 Jan 1970 00:00:00 +0000 (53 years ago)
〉"7fffffffffffffff" | into int -r 16 | into datetime
Fri, 11 Apr 2262 23:47:16 +0000 (in 239 years)
〉("7fffffffffffffff" | into int -r 16) * -1 | into datetime
Tue, 21 Sep 1677 00:12:43 +0000 (345 years ago)

`<date> | date to-record` and `<date> | date to-table`

Now both have a nanosecond field.

〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-record
╭────────────┬───────────╮
│ year       │ 2262      │
│ month      │ 4         │
│ day        │ 11        │
│ hour       │ 23        │
│ minute     │ 47        │
│ second     │ 16        │
│ nanosecond │ 854775807 │
│ timezone   │ +00:00    │
╰────────────┴───────────╯
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-table
╭───┬──────┬───────┬─────┬──────┬────────┬────────┬────────────┬──────────╮
│ # │ year │ month │ day │ hour │ minute │ second │ nanosecond │ timezone │
├───┼──────┼───────┼─────┼──────┼────────┼────────┼────────────┼──────────┤
│ 0 │ 2262 │     4 │  11 │   23 │     47 │     16 │  854775807 │ +00:00   │
╰───┴──────┴───────┴─────┴──────┴────────┴────────┴────────────┴──────────╯

This change was not mandated by the OP problem, but it is nice to be able to see the nanosecond bits that were present in Nushell date type all along.

Tests + Formatting

Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

cargo fmt --all -- --check to check standard code formatting (cargo fmt --all applies these changes)
cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect to check that you're using the standard code style
cargo test --workspace to check that all tests pass

After Submitting

If your PR had any user-facing changes, update the documentation after the PR is merged, if necessary. This will help us keep the docs up to date.

…nanosecond field

codecov · 2023-03-06T07:21:21Z

Codecov Report

Merging #8337 (7942257) into main (f93033c) will decrease coverage by 0.37%.
The diff coverage is 97.61%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8337      +/-   ##
==========================================
- Coverage   68.56%   68.20%   -0.37%     
==========================================
  Files         620      620              
  Lines       99304    99291      -13     
==========================================
- Hits        68086    67719     -367     
- Misses      31218    31572     +354

Impacted Files	Coverage Δ
crates/nu-command/src/conversions/into/int.rs	`78.57% <96.92%> (+2.81%)`	⬆️
crates/nu-command/src/conversions/into/datetime.rs	`86.41% <97.50%> (+1.93%)`	⬆️
crates/nu-command/src/date/to_record.rs	`94.20% <100.00%> (+0.30%)`	⬆️
crates/nu-command/src/date/to_table.rs	`90.90% <100.00%> (-2.53%)`	⬇️
crates/nu-protocol/src/value/mod.rs	`57.22% <100.00%> (+0.13%)`	⬆️
crates/nu-color-config/src/nu_style.rs	`9.31% <0.00%> (-70.92%)`	⬇️
crates/nu-protocol/src/ty.rs	`86.42% <0.00%> (-0.72%)`	⬇️
crates/nu-command/src/network/http/client.rs	`43.36% <0.00%> (-0.61%)`	⬇️
crates/nu-command/src/network/http/put.rs	`100.00% <0.00%> (ø)`
... and 9 more

fdncred · 2023-03-06T13:14:32Z

Nice work! Thanks for help with this. I'm happy to see more consistency.

bobhy · 2023-03-06T16:19:33Z

One other thing. In the course of developing this fix, I ran into a problem with dependency dtparse, which is used to parse datetime strings (but not literals) into type datetime. I've raised an issue with their developers, they're looking at it.

Here's details, probably should be noted in release notes...

known issues

You'll note that the sub-microsecond bits of a datetime are truncated when a string is parsed into a datetime.
If you calculate a datetime directly (e.g from a literal, date now or <datetime> + <duration> ), the precious nanosecond bits are preserved.

〉2023-03-02T01:02:03.987654321-05:00 | date to-record
╭────────────┬───────────╮
│ year       │ 2023      │
│ month      │ 3         │
│ day        │ 2         │
│ hour       │ 1         │
│ minute     │ 2         │
│ second     │ 3         │
│ nanosecond │ 987654321 │
│ timezone   │ -05:00    │
╰────────────┴───────────╯

〉'2023-03-02T01:02:03.987654321' | into datetime | date to-record
╭────────────┬───────────╮
│ year       │ 2023      │
│ month      │ 3         │
│ day        │ 2         │
│ hour       │ 1         │
│ minute     │ 2         │
│ second     │ 3         │
│ nanosecond │ 987654000 │
│ timezone   │ -05:00    │
╰────────────┴───────────╯

This is a limitation of dependency dtparse. See bspeice/dtparse#40

sholderbach · 2023-03-06T16:25:16Z

I think this is not too big of a deal as the system clocks also have a varying resolution and accuracy, typically much worse than a nanosecond (thanks side-channel attacks).

fdncred · 2023-03-07T23:31:55Z

@bobhy I'm wondering if I'm having some odd problem. I didn't expect nanos to always end in 000. Is that expected?

And the same with into int

date now | into int
1678231809663385000
date now | into int
1678231810551980000
date now | into int
1678231811351540000
date now | into int
1678231812148961000
date now | into int
1678231812916677000
date now | into int
1678231813737291000

# Description This PR fixes an error message that popped up after landing a PR #8337. I guess there were too many changes since the PR was submitted? # User-Facing Changes _(List of all changes that impact the user experience here. This helps us keep track of breaking changes.)_ # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date.

…ell#8337

#8805) # Description Pick up [enhancement in `dtparse`](bspeice/dtparse#41), to preserve nanoseconds in string-to-date conversions. # User-Facing Changes An especially eagle-eyed user might notice that prior versions of nushell were losing nanosecond precision when converting string format date/time to datetime. Before: ```nushell 〉'2023-03-02T01:02:03.987654321' | into datetime | date to-record ╭────────────┬───────────╮ │ year │ 2023 │ │ month │ 3 │ │ day │ 2 │ │ hour │ 1 │ │ minute │ 2 │ │ second │ 3 │ │ nanosecond │ 987654000 │ │ timezone │ -04:00 │ ╰────────────┴───────────╯ ``` Now, it just works(tm) ```nushell 〉'2023-03-02T01:02:03.987654321' | into datetime | date to-record ╭────────────┬───────────╮ │ year │ 2023 │ │ month │ 3 │ │ day │ 2 │ │ hour │ 1 │ │ minute │ 2 │ │ second │ 3 │ │ nanosecond │ 987654321 │ │ timezone │ -04:00 │ ╰────────────┴───────────╯ ``` # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [x] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [x] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - [x] `cargo test --workspace` to check that all tests pass Some (unrelated) tests did fail [^err] - [x] `cargo run -- crates/nu-std/tests.nu` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. # (Unrelated) test failures noted [^err]: Several tests are failing, unrelated to this PR ``` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` ```

…ell#8337 (nushell#8805) # Description Pick up [enhancement in `dtparse`](bspeice/dtparse#41), to preserve nanoseconds in string-to-date conversions. # User-Facing Changes An especially eagle-eyed user might notice that prior versions of nushell were losing nanosecond precision when converting string format date/time to datetime. Before: ```nushell 〉'2023-03-02T01:02:03.987654321' | into datetime | date to-record ╭────────────┬───────────╮ │ year │ 2023 │ │ month │ 3 │ │ day │ 2 │ │ hour │ 1 │ │ minute │ 2 │ │ second │ 3 │ │ nanosecond │ 987654000 │ │ timezone │ -04:00 │ ╰────────────┴───────────╯ ``` Now, it just works(tm) ```nushell 〉'2023-03-02T01:02:03.987654321' | into datetime | date to-record ╭────────────┬───────────╮ │ year │ 2023 │ │ month │ 3 │ │ day │ 2 │ │ hour │ 1 │ │ minute │ 2 │ │ second │ 3 │ │ nanosecond │ 987654321 │ │ timezone │ -04:00 │ ╰────────────┴───────────╯ ``` # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [x] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [x] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - [x] `cargo test --workspace` to check that all tests pass Some (unrelated) tests did fail [^err] - [x] `cargo run -- crates/nu-std/tests.nu` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` # After Submitting If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. # (Unrelated) test failures noted [^err]: Several tests are failing, unrelated to this PR ``` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` failures: ---- modules::module_import_env_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; use main.nu foo; foo · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:316:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- modules::module_import_env_2 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env main.nu; $env.FOO · ───┬─── · ╰── file not found ╰──── thread 'modules::module_import_env_2' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"foo"`', tests/modules/mod.rs:341:9 ---- overlays::overlay_use_do_cd_file_relative stdout ---- === stderr thread 'overlays::overlay_use_do_cd_file_relative' panicked at 'assertion failed: `(left == right)` left: `"nushell"`, right: `"test1"`', tests/overlays/mod.rs:919:9 ---- overlays::overlay_use_dont_cd_overlay stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env test1/test2/spam.nu; $env.PWD | path basename · ─────────┬───────── · ╰── file not found ╰──── thread 'overlays::overlay_use_dont_cd_overlay' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"overlay_use_dont_cd_overlay"`', tests/overlays/mod.rs:946:9 ---- parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol.nu; $env.FOO · ─────┬──── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"good"`', tests/parsing/mod.rs:177:9 ---- parsing::parse_file_relative_to_parsed_file_simple stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[source:1:1] 1 │ source-env lol/lol/lol.nu; $env.LOL · ───────┬────── · ╰── file not found ╰──── thread 'parsing::parse_file_relative_to_parsed_file_simple' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"lol"`', tests/parsing/mod.rs:99:9 ---- shell::const_nu_lib_dirs_relative stdout ---- === stderr Error: nu::shell::file_not_found × File not found ╭─[main.nu:1:1] 1 │ const NU_LIB_DIRS = [ 'scripts' ] 2 │ source-env foo.nu · ───┬── thread 'shell::const_nu_lib_dirs_relative' panicked at 'assertion failed: outcome.err.is_empty()', tests/shell/mod.rs:166:9 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"AndrásWithKitKat"`', tests/shell/pipeline/commands/external.rs:198:13 ---- shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings stdout ---- === stderr thread 'shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings' panicked at 'assertion failed: `(left == right)` left: `""`, right: `"jt_likes_cake.txt"`', tests/shell/pipeline/commands/external.rs:173:13 failures: modules::module_import_env_1 modules::module_import_env_2 overlays::overlay_use_do_cd_file_relative overlays::overlay_use_dont_cd_overlay parsing::parse_file_relative_to_parsed_file_dont_use_cwd_1 parsing::parse_file_relative_to_parsed_file_simple shell::const_nu_lib_dirs_relative shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_lines shell::pipeline::commands::external::it_evaluation::takes_rows_of_nu_value_strings test result: FAILED. 386 passed; 9 failed; 20 ignored; 0 measured; 0 filtered out; finished in 98.06s error: test failed, to rerun pass `-p nu --test main` ```

fdncred · 2023-08-19T14:29:31Z

@bobhy Do you have time to look at something? I believe this PR broke some things. In PR #10055, @amtoine and I thought it would be better to revert this PR. We were initially confused that into datetime -f some_format doesn't work. I was confused thinking it was formatting the output, when all along it was meant to format the input. So, then he puts up 10055 to revert that. We're noticing that you can't really do '20220604' | into datetime -f '%Y%m%d' because the logic of the code thinks that date is a timestamp and parses it incorrectly. Essentially, the logic never gets down to the --format part so it never runs. It would be great if you have time to look at this.

See #10055 for some ideas about how to fix it. I'm not sure what I'm suggesting is the right course of action.

bobhy · 2023-08-22T01:03:28Z

No, I traced the execution of `'20220604' | into datetime -f '%Y%m%d'` and found that the input string was properly fed into `chrono::DateTime<FixedOffset>::parse_from_string()`, but the latter was failing because the `--format` argument doesn't include a timezone token (`%z`).

…

On Sat, Aug 19, 2023 at 7:29 AM Darren Schroeder ***@***.***> wrote: @bobhy <https://github.com/bobhy> Do you have time to look at something? I believe this PR broke some things. In PR #10055 <#10055>, @amtoine <https://github.com/amtoine> and I thought it would be better to revert this PR. We were initially confused that into datetime -f some_format doesn't work. I was confused thinking it was formatting the output, when all along it was meant to format the input. So, then he puts up 10055 to revert that. We're noticing that you can't really do '20220604' | into datetime -f '%Y%m%d' because the logic of the code thinks that date is a timestamp and parses it incorrectly. Essentially, the logic never gets down to the --format part so it never runs. It would be great if you have time to look at this. — Reply to this email directly, view it on GitHub <#8337 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPBIZDE57RLF77IY26J5A3XWDEVLANCNFSM6AAAAAAVQXBAUI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bobhy added 7 commits March 4, 2023 21:49

date into int now in nanoseconds; first unit test works.

cfe9c7e

into int always stores nanoseconds; date to-record, to-table include …

356235a

…nanosecond field

wip into datetime --resolution flag

b72f1f1

datetime | into int now enforces date range

a18240a

address clippy warnings

0a5cd9c

correct typo in error message

f4181c6

Clippy getting snippy with the spacing.

7942257

sholderbach added the notes:breaking-changes This PR implies a change affecting users and has to be noted in the release notes label Mar 6, 2023

sholderbach linked an issue Mar 6, 2023 that may be closed by this pull request

spec and behavior of into datetime is ambiguous #8244

Closed

fdncred merged commit 2ad0fcb into nushell:main Mar 7, 2023

fdncred mentioned this pull request Mar 7, 2023

fixed an error message that popped up after landing #8356

Merged

bobhy deleted the fix_8244 branch April 7, 2023 12:42

bobhy added a commit to bobhy/nushell that referenced this pull request Apr 7, 2023

Pick up fix in dtparse for nanosec truncation noted at bottom of nush…

2d97f19

…ell#8337

bobhy mentioned this pull request Apr 27, 2023

Add -d flag to touch #8858

Closed

fdncred mentioned this pull request Aug 19, 2023

Revert "deprecate --format and --list in into datetime (#10017)" #10055

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 8244 -- store timestamps with nanosecond resolution (consistently)#8337

Fix 8244 -- store timestamps with nanosecond resolution (consistently)#8337
fdncred merged 7 commits intonushell:mainfrom
bobhy:fix_8244

bobhy commented Mar 6, 2023 •

edited

Loading

Uh oh!

codecov bot commented Mar 6, 2023 •

edited

Loading

Uh oh!

fdncred commented Mar 6, 2023

Uh oh!

bobhy commented Mar 6, 2023 •

edited

Loading

Uh oh!

sholderbach commented Mar 6, 2023

Uh oh!

fdncred commented Mar 7, 2023

Uh oh!

fdncred commented Aug 19, 2023 •

edited

Loading

Uh oh!

bobhy commented Aug 22, 2023 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bobhy commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

User-Facing Changes

<datetime> | into int

<int> | into datetime

<date> | date to-record and <date> | date to-table

Tests + Formatting

After Submitting

Uh oh!

codecov bot commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fdncred commented Mar 6, 2023

Uh oh!

bobhy commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

known issues

Uh oh!

sholderbach commented Mar 6, 2023

Uh oh!

fdncred commented Mar 7, 2023

Uh oh!

fdncred commented Aug 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobhy commented Aug 22, 2023 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bobhy commented Mar 6, 2023 •

edited

Loading

`<datetime> | into int`

`<int> | into datetime`

`<date> | date to-record` and `<date> | date to-table`

codecov bot commented Mar 6, 2023 •

edited

Loading

bobhy commented Mar 6, 2023 •

edited

Loading

fdncred commented Aug 19, 2023 •

edited

Loading