perf: parallelize per-item Plex watch-history reads during rule execution#2957
Merged
Conversation
…er-item abort - Collapse to one bounded-parallel layer in the comparator and revert the Plex-getter internal episode batching, so total in-flight lookups never exceed the cap (was up to cap x cap when nested). - Rename WATCH_HISTORY_CONCURRENCY -> RULE_EVALUATION_CONCURRENCY: the pool bounds per-item operand evaluation across all getters, not just Plex. - Restore abort checks inside the batched tasks for prompt cancellation.
This comment was marked as resolved.
This comment was marked as resolved.
Collaborator
Author
Contributor
|
Released to |
#2897 made getSeriesByTvdbId/getMovieByTmdbId uncached so the empty-show cleanup reads post-deletion truth (#2757/#2891) — but that also de-cached the rule-evaluation path, where the same series/movie resolves once per item, so episode-level rules over large libraries regressed badly. Add ArrLookupCache: a run-scoped memo created for the evaluation loop only and never handed to the collection/action phase, so the cleanup still reads fresh and 'delete then read stale cache' is structurally impossible. Only the two uncached arr identity lookups use it; Tautulli/Seerr/Plex/Jellyfin/ Emby already cache at the API layer.
Collaborator
Author
|
@daviddanko since we touch your bugfix in this PR, please confirm it's not reverted by this change. 🙏 |
…a a memo Guards the integration invariant behind the rule-evaluation ArrLookupCache: the cleanup path resolves the series from the uncached Sonarr client on every run. If a stale/memoized value ever leaked into this path, the second run (files now gone) would skip deletion — this test would fail.
Collaborator
Author
|
/release-pr |
Contributor
|
Released to |
After a quick test it looks fine! |
16 over-drove co-located CPU-heavy backends (Tautulli history queries on an all-in-one N100), pushing requests past the 10s timeout and triggering the retry feedback loop. 8 keeps concurrent load in check while staying far faster than the old sequential path.
Collaborator
Author
|
/release-pr |
Contributor
|
Released to |
Contributor
|
🎉 This PR is included in version 3.12.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up rule evaluation on large libraries, and fixes a recent Sonarr/Radarr lookup regression. No new dependencies.
1. Parallelize per-item operand resolution
Rules resolve each item's operand one at a time; for watch-history-style properties (no bulk endpoint, #2936) and per-item *arr lookups this dominates runtime. The comparator now resolves
firstVal/secondValin bounded parallel batches (chunk +Promise.all— the same idiom as the existing Plex collection-write batching), then runs the unchanged backward mutation loop.RULE_EVALUATION_CONCURRENCY = 8— the single global cap and the only batching layer (never nested inside getters), abort checked inside each task.2. Fix the #2897 Sonarr/Radarr lookup regression
#2897 made
getSeriesByTvdbId/getMovieByTmdbIduncached so the empty-show cleanup reads post-deletion truth (#2757 / #2891). Correct for the cleanup — but it also de-cached the rule-evaluation path, where the same series/movie resolves once per item, so episode-level rules over large libraries went from a handful of cached lookups to one uncached call per item per section.ArrLookupCacherestores the de-duplication without reintroducing the bug: a run-scoped memo created for the evaluation loop only and never handed to the collection/action phase, so the cleanup still reads fresh and a stale entry can't survive a deletion. The API-layer lookups stay uncached; only the two uncached *arr identity lookups use it (Tautulli/Seerr/Plex/Jellyfin/Emby already cache at the API layer).Real-world result
~13k-episode Tautulli+Sonarr rule on an all-in-one Intel N100:
8 is faster than 16 because removing the backend overload eliminated the timeout/retry stalls.
Tests:
ArrLookupCacheunit tests, an action-handler test asserting the cleanup reads series fresh (never via the memo), and the existing #2897 regression suites all pass.