Skip to content

perf: consolidate watch paths via ConsolidatingPathTrie on Windows#93

Merged
sapphi-red merged 7 commits into
mainfrom
perf/consolidate-paths-on-windows
May 20, 2026
Merged

perf: consolidate watch paths via ConsolidatingPathTrie on Windows#93
sapphi-red merged 7 commits into
mainfrom
perf/consolidate-paths-on-windows

Conversation

@sapphi-red

@sapphi-red sapphi-red commented May 19, 2026

Copy link
Copy Markdown
Member

Summary

On Windows, registering many sibling paths under the same parent previously opened one ReadDirectoryChangesW handle per child. This PR mirrors the fsevents consolidation pattern from #60: when 10 or more sibling paths are registered under the same parent, they collapse into a single watch on the parent instead.

closes #62

How it works

  • UserWatch struct. self.watches now stores the user's intent plus the resolved OS-level primary dir (and optional tracked-parent for TargetMode::TrackPath), instead of a bare WatchMode.
  • rebuild_watch_handles. Builds a ConsolidatingPathTrie over the primary dirs of all user watches, computes the recursive flag per consolidated target by checking for strict descendants, layers in tracked-parent watches, and diffs against the open handles to converge OS state. add_watch, remove_watch, and apply_staged now update self.watches and call this, rather than poking add_watch_raw directly.
  • is_event_covered. Since consolidation can leave the OS dir different from any user request, event filtering now walks ancestors instead of doing a direct hash lookup.
  • Consolidated pre-open. apply_staged no longer does a per-path pre_open_tracked_parent, which caused N opens that consolidation usually threw away. A single pre_open_consolidated pass runs the same consolidation trie over staged parents and opens the consolidated set directly, so the later rebuild_watch_handles usually finds handles already in place, with no close-and-reopen. This preserves the Windows metadata quirk fix: a directory's metadata() must run after its parent is watched, or the parent's later FILE_ACTION_MODIFIED is silently dropped.

Benchmarks

A new windows_sibling_subdirs workload watches N sibling directories under a shared parent. This is the shape issue #62 describes, for example per-package or per-dependency directories in a monorepo. The existing windows_paths_mut bench hid consolidation's benefit because 40% of its files sit directly in the tempdir root, collapsing everything into one recursive watch.

The sibling_subdirs workload was also added for all watcher backends (inotify, fsevents, kqueue, ReadDirectoryChangesW, poll). Existing paths_mut file counts were lowered from [100, 500, 1000] to [100, 200, 400] to keep runtimes reasonable.

Results

  • windows_sibling_subdirs runs 27 to 80% faster with consolidation, scaling with N.
  • The consolidated pre-open pass brings the windows_paths_mut workload back to par with the no-consolidation baseline (nonrecursive/100: 2.60 ms to 2.09 ms, 20% faster), while windows_sibling_subdirs/nonrecursive/500 improved from 11.92 ms to 9.25 ms (22% faster).

@sapphi-red sapphi-red changed the title perf: consolidate watch paths via ConsolidatingPathTrie perf: consolidate watch paths via ConsolidatingPathTrie May 19, 2026
@sapphi-red sapphi-red force-pushed the perf/consolidate-paths-on-windows branch 2 times, most recently from 236b6c8 to d91c372 Compare May 19, 2026 10:06
Close #62.

Mirror the fsevents PR #60 pattern: when 10+ sibling paths are registered
under the same parent, collapse them into a single watch on the parent
instead of opening one `ReadDirectoryChangesW` handle per child.

Implementation:
* Replace the `WatchMode` value of `self.watches` with a new `UserWatch`
  struct that carries the user's intent plus the resolved OS-level
  primary dir (and optional tracked-parent for `TargetMode::TrackPath`).
* Add `rebuild_watch_handles`: build a `ConsolidatingPathTrie` over the
  primary dirs from every user watch, compute the recursive flag per
  consolidated target by checking for strict descendants in the user
  watches, layer in tracked-parent watches that aren't already covered,
  and diff against the open `watch_handles` to converge the OS state.
* `add_watch` / `remove_watch` / `apply_staged` now update `self.watches`
  first and then call `rebuild_watch_handles` instead of poking
  `add_watch_raw` directly.
* The event filter at `handle_event` was a direct hash lookup on the OS
  dir name; consolidation can leave the OS dir different from any user
  request, so it now walks ancestors via `is_event_covered`.
* `pre_open_tracked_parent` works around a Windows quirk where calling
  `path.metadata()` on a directory *before* its parent dir has a
  `ReadDirectoryChangesW` handle silently drops the parent's later
  `FILE_ACTION_MODIFIED` event for that directory. The pre-call opens
  the tracked parent before any metadata is read, matching the order
  the previous per-call code used.

Adds five new tests covering: 10-sibling consolidation, event delivery
through a consolidated parent, mixed recursive/non-recursive collapse,
de-consolidation when the count drops back below the threshold, and the
batched `paths_mut().commit()` path.
@sapphi-red sapphi-red force-pushed the perf/consolidate-paths-on-windows branch from d91c372 to 22d450b Compare May 19, 2026 10:52
The existing `windows_paths_mut` bench puts 40% of its files directly in
the tempdir root, so every watched parent dir consolidates back into
that root and the result is a single recursive watch on the entire
tempdir subtree. That hides consolidation's actual benefit — on this
workload consolidation is slightly slower because pre-opened handles
are immediately replaced.

The new `windows_sibling_subdirs` bench watches N sibling directories
under a shared parent, which is the shape issue #62 is about (e.g.,
watching each package directory in a monorepo, or each dependency
directory). With consolidation enabled this workload is 27–80% faster
than with consolidation disabled, with the win scaling as N grows.
@sapphi-red sapphi-red force-pushed the perf/consolidate-paths-on-windows branch from 8d4ab77 to bc2a0b2 Compare May 19, 2026 11:35
…ed commit

`apply_staged` previously did a per-path `pre_open_tracked_parent` —
N opens just to satisfy the Windows metadata-quirk that requires the
direct parent to be watched before `path.metadata()` runs. Whenever
consolidation later collapsed those N parents into a higher ancestor,
all the pre-opens were thrown away (N closes + 1 reopen).

Replace the per-path pre-open with a single `pre_open_consolidated`
step that runs the same consolidation trie lexically over each staged
path's direct parent, opens the *consolidated* set, and picks the
recursive flag based on whether any staged parent is a strict
descendant of the consolidated target. The subsequent
`rebuild_watch_handles` against the actual primaries usually finds
the target already in place, so no close-and-reopen is needed.

On the existing `windows_paths_mut` bench, which collapses everything
into a single recursive watch on the test root because 40% of files
sit directly in it, this drops setup from ~13 OS ops to ~1. Measured
delta on `windows_paths_mut/nonrecursive/1000`: 21.37 ms → 20.78 ms
(-3%) and `windows_paths_mut/nonrecursive/100`: 2.60 ms → 2.09 ms
(-20%), bringing the consolidated path back to par with the
no-consolidation baseline while keeping the consolidation benefit.

The sibling-subdir workload — where consolidation actually saves
handles — also speeds up at the high end (`windows_sibling_subdirs/
nonrecursive/500`: 11.92 ms → 9.25 ms, -22%).

All 87 existing tests pass unchanged; the conservative open uses
exactly the same direct-parent invariant that `pre_open_tracked_parent`
relied on, so the metadata-quirk fix is preserved.
Add a `sibling_subdirs` workload to the paths_mut benchmark for every
watcher backend (inotify, fsevents, kqueue, ReadDirectoryChangesW, and
the cross-platform poll watcher). It watches many sibling subdirectories
under a shared parent -- representative of monorepo or dependency-directory
monitoring -- to measure the benefit of path consolidation.

Also lower the existing paths_mut file counts from [100, 500, 1000] to
[100, 200, 400] to keep benchmark runtimes reasonable.
`watches` previously stored a `UserWatch` struct that bundled the user's
raw `WatchMode` request together with the metadata-resolved coverage
(`primary` dir, `tracked_parent`). This conflated the source of truth
with derived state.

Split it into two maps keyed by user path:
* `watches: HashMap<PathBuf, WatchMode>` keeps only the raw user request.
* `resolved_watches: HashMap<PathBuf, ResolvedWatch>` holds the resolved
  coverage. It is server-only and not shared with the event thread, which
  needs only the raw mode.

`rebuild_watch_handles` prunes `resolved_watches` against `watches` at the
start, since the event thread can drop a `NoTrack` entry from `watches`
directly without touching `resolved_watches`. This keeps consolidation
from ever using a stale resolution.

All 87 lib tests pass.
@sapphi-red sapphi-red changed the title perf: consolidate watch paths via ConsolidatingPathTrie perf: consolidate watch paths via ConsolidatingPathTrie on Windows May 20, 2026
@sapphi-red sapphi-red marked this pull request as ready for review May 20, 2026 07:38
@sapphi-red sapphi-red merged commit 32a09b5 into main May 20, 2026
12 checks passed
@sapphi-red sapphi-red deleted the perf/consolidate-paths-on-windows branch May 20, 2026 07:38
@rolldown-guard rolldown-guard Bot mentioned this pull request May 19, 2026
sapphi-red pushed a commit that referenced this pull request May 20, 2026
## 🤖 New release

* `rolldown-notify`: 10.3.1 -> 10.4.0 (✓ API compatible changes)
* `rolldown-notify-debouncer-mini`: 0.8.7 -> 0.8.8
* `rolldown-notify-debouncer-full`: 0.7.7 -> 0.7.8

<details><summary><i><b>Changelog</b></i></summary><p>

## `rolldown-notify`

<blockquote>

##
[10.4.0](rolldown-notify-v10.3.1...rolldown-notify-v10.4.0)
- 2026-05-20

### Added

- consolidate paths further more for fsevents backend if needed
([#91](#91))

### Other

- consolidate watch paths via `ConsolidatingPathTrie` on Windows
([#93](#93))
- batch path changes via `PathsMut` on Windows
([#94](#94))
</blockquote>

## `rolldown-notify-debouncer-mini`

<blockquote>

##
[0.8.8](rolldown-notify-debouncer-mini-v0.8.7...rolldown-notify-debouncer-mini-v0.8.8)
- 2026-05-20

### Other

- updated the following local packages: rolldown-notify
</blockquote>

## `rolldown-notify-debouncer-full`

<blockquote>

##
[0.7.8](rolldown-notify-debouncer-full-v0.7.7...rolldown-notify-debouncer-full-v0.7.8)
- 2026-05-20

### Other

- updated the following local packages: rolldown-notify
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: rolldown-guard[bot] <278280044+rolldown-guard[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve windows backend performance by consolidating watch paths

1 participant