Conversation
…ed executor - Remove shared _executor ClassVar and get_executor() method - Use per-call ThreadPoolExecutor with context manager for automatic cleanup - Add DDGS.threads cap to limit max_workers per search - Fix bug: log f_engine.name instead of engine.name in exception handler - Prevents thread pool starvation under concurrent load
Owner
Author
shenjianan97
added a commit
to shenjianan97/persistent-agent-runtime
that referenced
this pull request
Apr 16, 2026
The previous threading.Lock serialized every web_search call to work around a thread pool starvation bug in ddgs < 9.12.1, where a process- wide ClassVar ThreadPoolExecutor was shared across all DDGS instances. Under concurrent callers the shared pool saturated, and combined with DDG's per-IP rate limiting this presented as "blocked forever" (see commit 310be71 which introduced the lock). ddgs 9.12.1 (upstream PR deedy5/ddgs#445) replaced the shared pool with a per-call, context-managed ThreadPoolExecutor. Each search now owns an isolated, bounded executor that shuts down on exit, so inter-call pool contention is gone. - Pin ddgs>=9.12.1 so we can't slide back into the broken range - Drop threading.Lock and thread import - Introduce asyncio.Semaphore(max_concurrent=3) on the async side: lets up to 3 searches run in parallel via asyncio.to_thread while still capping fan-out to stay under DDG's per-IP rate limit - max_concurrent is a constructor knob so the tuning can move without a code change when we learn more from production load Adds 4 unit tests verifying the semaphore actually caps in-flight work, parallel wall-clock beats serial, and the threading.Lock is gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary