-
Notifications
You must be signed in to change notification settings - Fork 2.1k
bytes at gets slower the bigger the input is #13668
Copy link
Copy link
Open
Labels
A:streamingIssues related to streaming data (or collecting data when it should be streamed)Issues related to streaming data (or collecting data when it should be streamed)performanceWork to make nushell quicker and use less resourcesWork to make nushell quicker and use less resourcesstatus:needs-triageAn issue that hasn't had any proper lookAn issue that hasn't had any proper look
Description
Describe the bug
I'm currently writing a command that performs an action every 2048 bytes of binary data. That's when I noticed that the command takes longer, the bigger the input data is. This sounds like the binary data is first loaded entirely and then sliced. Taking the first 2048 bytes of data should always take the same amount of time, regardless of input size.
How to reproduce
- Create a file that is
500 KB - Create a file that is
500 MB let small_input = (open --raw 500kb.bin | into binary)let big_input = (open --raw 500mb.bin | into binary)timeit { $small_input | bytes at ..2048 }timeit { $big_input | bytes at ..2048 }--> will take around 1000 times longer
Expected behavior
Should always take the same amount of time. Comparable tools work like this:
timeit { head -c 2048 500kb.bin }andtimeit { head -c 2048 500mb.bin }timeit { od -j 20000 -N 2048 500kb.bin }andtimeit { od -j 20000 -N 2048 500mb.bin }(get 2048 bytes starting from byte 20000)
These commands don't take longer to execute if the input is bigger in size. Even slices from the end of the file don't take longer.
Screenshots
No response
Configuration
| key | value |
|---|---|
| version | 0.96.1 |
| major | 0 |
| minor | 96 |
| patch | 1 |
| branch | |
| commit_hash | |
| build_os | linux-x86_64 |
| build_target | x86_64-unknown-linux-gnu |
| rust_version | rustc 1.79.0 (129f3b996 2024-06-10) (gentoo) |
| cargo_version | cargo 1.79.0 |
| build_time | 2024-08-12 09:18:31 +02:00 |
| build_rust_channel | release |
| allocator | mimalloc |
| features | default, sqlite, system-clipboard, trash |
| installed_plugins | bin_reader |
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A:streamingIssues related to streaming data (or collecting data when it should be streamed)Issues related to streaming data (or collecting data when it should be streamed)performanceWork to make nushell quicker and use less resourcesWork to make nushell quicker and use less resourcesstatus:needs-triageAn issue that hasn't had any proper lookAn issue that hasn't had any proper look