Skip to content

perf: consolidate paths to parent path when threshold exceeded for fsevents watcher#60

Merged
sapphi-red merged 1 commit into
mainfrom
12-19-perf_consolidate_paths_to_parent_path_when_threshold_exceeded_for_fsevents_watcher
Dec 19, 2025
Merged

perf: consolidate paths to parent path when threshold exceeded for fsevents watcher#60
sapphi-red merged 1 commit into
mainfrom
12-19-perf_consolidate_paths_to_parent_path_when_threshold_exceeded_for_fsevents_watcher

Conversation

@sapphi-red

Copy link
Copy Markdown
Member

If more than 10 paths are watched, watch the parent path instead.
This behavior was inspired by chokidar:
https://github.com/paulmillr/chokidar/blob/7c50e25d10a497ce4409f6e52eb630f0d7647b97/lib/fsevents-handler.js#L113-L119

@sapphi-red sapphi-red merged commit ca91535 into main Dec 19, 2025
9 checks passed
@sapphi-red sapphi-red deleted the 12-19-perf_consolidate_paths_to_parent_path_when_threshold_exceeded_for_fsevents_watcher branch December 19, 2025 11:20
@Boshen Boshen mentioned this pull request Dec 19, 2025
graphite-app Bot pushed a commit to rolldown/rolldown that referenced this pull request Dec 22, 2025
Now that I implemented some improvements (rolldown/notify#59, rolldown/notify#60), it doesn't error on large projects. kqueue doesn't have a way to watch recursively, so it's faster to use fsevent which supports recursive watches.
sapphi-red added a commit that referenced this pull request May 19, 2026
Close #62.

Mirror the fsevents PR #60 pattern: when 10+ sibling paths are registered
under the same parent, collapse them into a single watch on the parent
instead of opening one `ReadDirectoryChangesW` handle per child.

Implementation:
* Replace the `WatchMode` value of `self.watches` with a new `UserWatch`
  struct that carries the user's intent plus the resolved OS-level
  primary dir (and optional tracked-parent for `TargetMode::TrackPath`).
* Add `rebuild_watch_handles`: build a `ConsolidatingPathTrie` over the
  primary dirs from every user watch, compute the recursive flag per
  consolidated target by checking for strict descendants in the user
  watches, layer in tracked-parent watches that aren't already covered,
  and diff against the open `watch_handles` to converge the OS state.
* `add_watch` / `remove_watch` / `apply_staged` now update `self.watches`
  first and then call `rebuild_watch_handles` instead of poking
  `add_watch_raw` directly.
* The event filter at `handle_event` was a direct hash lookup on the OS
  dir name; consolidation can leave the OS dir different from any user
  request, so it now walks ancestors via `is_event_covered`.
* `pre_open_tracked_parent` works around a Windows quirk where calling
  `path.metadata()` on a directory *before* its parent dir has a
  `ReadDirectoryChangesW` handle silently drops the parent's later
  `FILE_ACTION_MODIFIED` event for that directory. The pre-call opens
  the tracked parent before any metadata is read, matching the order
  the previous per-call code used.

Adds five new tests covering: 10-sibling consolidation, event delivery
through a consolidated parent, mixed recursive/non-recursive collapse,
de-consolidation when the count drops back below the threshold, and the
batched `paths_mut().commit()` path.
sapphi-red added a commit that referenced this pull request May 19, 2026
Close #62.

Mirror the fsevents PR #60 pattern: when 10+ sibling paths are registered
under the same parent, collapse them into a single watch on the parent
instead of opening one `ReadDirectoryChangesW` handle per child.

Implementation:
* Replace the `WatchMode` value of `self.watches` with a new `UserWatch`
  struct that carries the user's intent plus the resolved OS-level
  primary dir (and optional tracked-parent for `TargetMode::TrackPath`).
* Add `rebuild_watch_handles`: build a `ConsolidatingPathTrie` over the
  primary dirs from every user watch, compute the recursive flag per
  consolidated target by checking for strict descendants in the user
  watches, layer in tracked-parent watches that aren't already covered,
  and diff against the open `watch_handles` to converge the OS state.
* `add_watch` / `remove_watch` / `apply_staged` now update `self.watches`
  first and then call `rebuild_watch_handles` instead of poking
  `add_watch_raw` directly.
* The event filter at `handle_event` was a direct hash lookup on the OS
  dir name; consolidation can leave the OS dir different from any user
  request, so it now walks ancestors via `is_event_covered`.
* `pre_open_tracked_parent` works around a Windows quirk where calling
  `path.metadata()` on a directory *before* its parent dir has a
  `ReadDirectoryChangesW` handle silently drops the parent's later
  `FILE_ACTION_MODIFIED` event for that directory. The pre-call opens
  the tracked parent before any metadata is read, matching the order
  the previous per-call code used.

Adds five new tests covering: 10-sibling consolidation, event delivery
through a consolidated parent, mixed recursive/non-recursive collapse,
de-consolidation when the count drops back below the threshold, and the
batched `paths_mut().commit()` path.
sapphi-red added a commit that referenced this pull request May 19, 2026
Close #62.

Mirror the fsevents PR #60 pattern: when 10+ sibling paths are registered
under the same parent, collapse them into a single watch on the parent
instead of opening one `ReadDirectoryChangesW` handle per child.

Implementation:
* Replace the `WatchMode` value of `self.watches` with a new `UserWatch`
  struct that carries the user's intent plus the resolved OS-level
  primary dir (and optional tracked-parent for `TargetMode::TrackPath`).
* Add `rebuild_watch_handles`: build a `ConsolidatingPathTrie` over the
  primary dirs from every user watch, compute the recursive flag per
  consolidated target by checking for strict descendants in the user
  watches, layer in tracked-parent watches that aren't already covered,
  and diff against the open `watch_handles` to converge the OS state.
* `add_watch` / `remove_watch` / `apply_staged` now update `self.watches`
  first and then call `rebuild_watch_handles` instead of poking
  `add_watch_raw` directly.
* The event filter at `handle_event` was a direct hash lookup on the OS
  dir name; consolidation can leave the OS dir different from any user
  request, so it now walks ancestors via `is_event_covered`.
* `pre_open_tracked_parent` works around a Windows quirk where calling
  `path.metadata()` on a directory *before* its parent dir has a
  `ReadDirectoryChangesW` handle silently drops the parent's later
  `FILE_ACTION_MODIFIED` event for that directory. The pre-call opens
  the tracked parent before any metadata is read, matching the order
  the previous per-call code used.

Adds five new tests covering: 10-sibling consolidation, event delivery
through a consolidated parent, mixed recursive/non-recursive collapse,
de-consolidation when the count drops back below the threshold, and the
batched `paths_mut().commit()` path.
sapphi-red added a commit that referenced this pull request May 20, 2026
…93)

### Summary

On Windows, registering many sibling paths under the same parent
previously opened one `ReadDirectoryChangesW` handle per child. This PR
mirrors the fsevents consolidation pattern from #60: when 10 or more
sibling paths are registered under the same parent, they collapse into a
single watch on the parent instead.

closes #62

### How it works

* `UserWatch` struct. `self.watches` now stores the user's intent plus
the resolved OS-level primary dir (and optional tracked-parent for
`TargetMode::TrackPath`), instead of a bare `WatchMode`.
* `rebuild_watch_handles`. Builds a `ConsolidatingPathTrie` over the
primary dirs of all user watches, computes the recursive flag per
consolidated target by checking for strict descendants, layers in
tracked-parent watches, and diffs against the open handles to converge
OS state. `add_watch`, `remove_watch`, and `apply_staged` now update
`self.watches` and call this, rather than poking `add_watch_raw`
directly.
* `is_event_covered`. Since consolidation can leave the OS dir different
from any user request, event filtering now walks ancestors instead of
doing a direct hash lookup.
* Consolidated pre-open. `apply_staged` no longer does a per-path
`pre_open_tracked_parent`, which caused N opens that consolidation
usually threw away. A single `pre_open_consolidated` pass runs the same
consolidation trie over staged parents and opens the consolidated set
directly, so the later `rebuild_watch_handles` usually finds handles
already in place, with no close-and-reopen. This preserves the Windows
metadata quirk fix: a directory's `metadata()` must run after its parent
is watched, or the parent's later `FILE_ACTION_MODIFIED` is silently
dropped.

### Benchmarks

A new `windows_sibling_subdirs` workload watches N sibling directories
under a shared parent. This is the shape issue #62 describes, for
example per-package or per-dependency directories in a monorepo. The
existing `windows_paths_mut` bench hid consolidation's benefit because
40% of its files sit directly in the tempdir root, collapsing everything
into one recursive watch.

The `sibling_subdirs` workload was also added for all watcher backends
(inotify, fsevents, kqueue, `ReadDirectoryChangesW`, poll). Existing
`paths_mut` file counts were lowered from `[100, 500, 1000]` to `[100,
200, 400]` to keep runtimes reasonable.

### Results

* `windows_sibling_subdirs` runs 27 to 80% faster with consolidation,
scaling with N.
* The consolidated pre-open pass brings the `windows_paths_mut` workload
back to par with the no-consolidation baseline (`nonrecursive/100`: 2.60
ms to 2.09 ms, 20% faster), while
`windows_sibling_subdirs/nonrecursive/500` improved from 11.92 ms to
9.25 ms (22% faster).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant