Skip to content

perf: parallelize per-item Plex watch-history reads during rule execution#2957

Merged
enoch85 merged 9 commits into
developmentfrom
perf/parallel-plex-watch-history
May 22, 2026
Merged

perf: parallelize per-item Plex watch-history reads during rule execution#2957
enoch85 merged 9 commits into
developmentfrom
perf/parallel-plex-watch-history

Conversation

@enoch85

@enoch85 enoch85 commented May 21, 2026

Copy link
Copy Markdown
Collaborator

Speeds up rule evaluation on large libraries, and fixes a recent Sonarr/Radarr lookup regression. No new dependencies.

1. Parallelize per-item operand resolution

Rules resolve each item's operand one at a time; for watch-history-style properties (no bulk endpoint, #2936) and per-item *arr lookups this dominates runtime. The comparator now resolves firstVal/secondVal in bounded parallel batches (chunk + Promise.all — the same idiom as the existing Plex collection-write batching), then runs the unchanged backward mutation loop.

  • RULE_EVALUATION_CONCURRENCY = 8 — the single global cap and the only batching layer (never nested inside getters), abort checked inside each task.
  • Conservative by design: the binding constraint is the slowest co-located backend (e.g. Tautulli's CPU-heavy history queries), not host core count. Higher values over-drive such backends past their request timeout, which then retries and amplifies load.

2. Fix the #2897 Sonarr/Radarr lookup regression

#2897 made getSeriesByTvdbId/getMovieByTmdbId uncached so the empty-show cleanup reads post-deletion truth (#2757 / #2891). Correct for the cleanup — but it also de-cached the rule-evaluation path, where the same series/movie resolves once per item, so episode-level rules over large libraries went from a handful of cached lookups to one uncached call per item per section.

ArrLookupCache restores the de-duplication without reintroducing the bug: a run-scoped memo created for the evaluation loop only and never handed to the collection/action phase, so the cleanup still reads fresh and a stale entry can't survive a deletion. The API-layer lookups stay uncached; only the two uncached *arr identity lookups use it (Tautulli/Seerr/Plex/Jellyfin/Emby already cache at the API layer).

Real-world result

~13k-episode Tautulli+Sonarr rule on an all-in-one Intel N100:

Runtime CPU Tautulli errors
Before (stable, sequential) hours
Concurrency 16 28 min 90–95% many (timeout → retry storm → items skipped)
Concurrency 8 (this PR) 18 min 60–85% none

8 is faster than 16 because removing the backend overload eliminated the timeout/retry stalls.

Tests: ArrLookupCache unit tests, an action-handler test asserting the cleanup reads series fresh (never via the memo), and the existing #2897 regression suites all pass.

enoch85 added 3 commits May 21, 2026 20:24
…er-item abort

- Collapse to one bounded-parallel layer in the comparator and revert the
  Plex-getter internal episode batching, so total in-flight lookups never
  exceed the cap (was up to cap x cap when nested).
- Rename WATCH_HISTORY_CONCURRENCY -> RULE_EVALUATION_CONCURRENCY: the pool
  bounds per-item operand evaluation across all getters, not just Plex.
- Restore abort checks inside the batched tasks for prompt cancellation.
@enoch85

This comment was marked as resolved.

@enoch85

enoch85 commented May 21, 2026

Copy link
Copy Markdown
Collaborator Author

@github-actions

Copy link
Copy Markdown
Contributor

Released to maintainerr/maintainerr:pr-2957 🚀

@SmolSoftBoi SmolSoftBoi added the enhancement New feature or request label May 21, 2026
#2897 made getSeriesByTvdbId/getMovieByTmdbId uncached so the empty-show
cleanup reads post-deletion truth (#2757/#2891) — but that also de-cached
the rule-evaluation path, where the same series/movie resolves once per
item, so episode-level rules over large libraries regressed badly.

Add ArrLookupCache: a run-scoped memo created for the evaluation loop only
and never handed to the collection/action phase, so the cleanup still reads
fresh and 'delete then read stale cache' is structurally impossible. Only
the two uncached arr identity lookups use it; Tautulli/Seerr/Plex/Jellyfin/
Emby already cache at the API layer.
@enoch85

enoch85 commented May 21, 2026

Copy link
Copy Markdown
Collaborator Author

@daviddanko since we touch your bugfix in this PR, please confirm it's not reverted by this change. 🙏

enoch85 added 2 commits May 21, 2026 21:45
…a a memo

Guards the integration invariant behind the rule-evaluation ArrLookupCache:
the cleanup path resolves the series from the uncached Sonarr client on every
run. If a stale/memoized value ever leaked into this path, the second run
(files now gone) would skip deletion — this test would fail.
@enoch85

enoch85 commented May 21, 2026

Copy link
Copy Markdown
Collaborator Author

/release-pr

@github-actions

Copy link
Copy Markdown
Contributor

Released to maintainerr/maintainerr:pr-2957 🚀

@daviddanko

Copy link
Copy Markdown

@daviddanko since we touch your bugfix in this PR, please confirm it's not reverted by this change. 🙏

After a quick test it looks fine!

enoch85 added 3 commits May 22, 2026 15:23
16 over-drove co-located CPU-heavy backends (Tautulli history queries on an
all-in-one N100), pushing requests past the 10s timeout and triggering the
retry feedback loop. 8 keeps concurrent load in check while staying far
faster than the old sequential path.
@enoch85

enoch85 commented May 22, 2026

Copy link
Copy Markdown
Collaborator Author

/release-pr

@github-actions

Copy link
Copy Markdown
Contributor

Released to maintainerr/maintainerr:pr-2957 🚀

@enoch85 enoch85 merged commit 656c9e6 into development May 22, 2026
14 checks passed
@enoch85 enoch85 deleted the perf/parallel-plex-watch-history branch May 22, 2026 16:47
@maintainerr-automation

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 3.12.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@enoch85 enoch85 added this to the 3.12.1 milestone May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants