Skip to content

Large scale watching of paths appears to drop/miss events #412

@chipsenkbeil

Description

@chipsenkbeil

System details

  • Rust version (if building from source): rustc --version: rustc 1.60.0 (7737e0b5c 2022-04-04)
  • Notify version (or commit hash if building from git): 5.0.0-pre.15

Mac

M1 Macbook Air from 2020.

image

Windows

Traditional Windows 11 machine.

image

Linux

Running Alpine Linux using Mac Parallels on an M1 Mac, which means Alpine is using ARM.

image

What you did (as detailed as you can)

Example repo where you can run test: https://github.com/chipsenkbeil/notify-stress-test

  1. Create a watcher using notify::recommended_watcher
  2. Have events get sent from the watcher handler out of the thread using std::sync::mpsc::Sender
  3. Continually receive events on a separate thread using std::sync::mpsc::Receiver
  4. Create a large number of paths to watch (files or directories) and watch each path using the same watcher
  5. Perform something within each path (modify a file, add a file to a directory, etc)
  6. Tick off each path with an event received from the std::sync::mpsc::Receiver

What you expected

Path modifications or other events show up for watched paths at scale. For example, if watching 1500 individual files, modifying each file would result in some event being captured and passed along.

What happened

With enough different paths being watched, events start to be missing or dropped. For example, watching 1500 individual files, modifying each file, 246 paths were never reported as modified.

❯ cargo test
   Compiling notify-stress-test v0.1.0 (/Users/senkwich/projects/notify-stress-test)
    Finished test [unoptimized + debuginfo] target(s) in 0.80s
     Running unittests (target/debug/deps/notify_stress_test-af3191235b8b1ce0)

running 1 test
test tests::stress_test ... FAILED

failures:

---- tests::stress_test stdout ----
thread 'tests::stress_test' panicked at 'assertion failed: `(left == right)`
  left: `246`,
 right: `0`: 246/1500 file paths not modified', src/lib.rs:88:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::stress_test

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 18.16s

error: test failed, to rerun pass '--lib'

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions