Skip to content

bytes at gets slower the bigger the input is #13668

@freijon

Description

@freijon

Describe the bug

I'm currently writing a command that performs an action every 2048 bytes of binary data. That's when I noticed that the command takes longer, the bigger the input data is. This sounds like the binary data is first loaded entirely and then sliced. Taking the first 2048 bytes of data should always take the same amount of time, regardless of input size.

How to reproduce

  1. Create a file that is 500 KB
  2. Create a file that is 500 MB
  3. let small_input = (open --raw 500kb.bin | into binary)
  4. let big_input = (open --raw 500mb.bin | into binary)
  5. timeit { $small_input | bytes at ..2048 }
  6. timeit { $big_input | bytes at ..2048 } --> will take around 1000 times longer

Expected behavior

Should always take the same amount of time. Comparable tools work like this:

  • timeit { head -c 2048 500kb.bin } and timeit { head -c 2048 500mb.bin }
  • timeit { od -j 20000 -N 2048 500kb.bin } and timeit { od -j 20000 -N 2048 500mb.bin } (get 2048 bytes starting from byte 20000)

These commands don't take longer to execute if the input is bigger in size. Even slices from the end of the file don't take longer.

Screenshots

No response

Configuration

key value
version 0.96.1
major 0
minor 96
patch 1
branch
commit_hash
build_os linux-x86_64
build_target x86_64-unknown-linux-gnu
rust_version rustc 1.79.0 (129f3b996 2024-06-10) (gentoo)
cargo_version cargo 1.79.0
build_time 2024-08-12 09:18:31 +02:00
build_rust_channel release
allocator mimalloc
features default, sqlite, system-clipboard, trash
installed_plugins bin_reader

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    A:streamingIssues related to streaming data (or collecting data when it should be streamed)performanceWork to make nushell quicker and use less resourcesstatus:needs-triageAn issue that hasn't had any proper look

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions