What does this PR do?

Fixes a resource leak where MCP stdio subprocesses (e.g. mempalace.mcp_server, any stdio-configured MCP server) accumulate as orphan session leaders whenever the SDK's subprocess teardown fails — most commonly when a cron job is cancelled, times out, or raises mid-flight.

Root cause: stdio_client spawns the MCP server child via start_new_session=True (for killpg() support), which makes it a session leader (Ssl in ps, PID == PGID == SID). When the enclosing asyncio task is cancelled or the context exits with an exception, the SDK sometimes fails to reap the child before the async generator finalizes. The surviving subprocess escapes the gateway's process group / cgroup tree, so neither systemd (KillMode=mixed and KillMode=control-group on soft restarts) nor _stop_mcp_loop() at shutdown can reliably kill it. Every cron tick that touches an MCP tool leaks one more server, each holding ~25–100 MB.

Fix — v2 (parallel-safe, rebased on #13021):

Previous iterations of this PR called _kill_orphaned_mcp_children() from run_job()'s finally: block, clearing the full _stdio_pids set. That races with:

• concurrent cron jobs after fix(cron): run due jobs in parallel to prevent serial tick starvation #13021 made tick() use a ThreadPoolExecutor (a job finishing first would kill the MCP servers of its still-running siblings);
• live user chat sessions (Telegram/Slack) whose MCP calls share the same tracking set.

This revision separates active from orphaned PIDs at the source:

1. tools/mcp_tool.py — _run_stdio now wraps the stdio_client + ClientSession contexts in a try/finally. On any exit path (clean, exception, cancellation) PIDs are removed from _stdio_pids; if a PID is still alive (probed via os.kill(pid, 0)) it is moved into a new _orphan_stdio_pids set. Liveness is the signal, not the code path — so we catch cancellation-based leaks without any except CancelledError plumbing. Upstream now tracks each stdio child as pid -> server name and redirects server stderr to the shared log; that stays intact, integrated with the try/finally + orphan set.
2. tools/mcp_tool.py — _kill_orphaned_mcp_children(include_active: bool = False). The default now reaps only the orphan set, so concurrent sessions are never disturbed. The existing _stop_mcp_loop() call passes include_active=True to preserve the previous "kill everything" semantics after the loop has stopped. Reaping uses the same graceful path as upstream (SIGTERM, short wait, SIGKILL) when clearing worklists.
3. cron/scheduler.py — the cleanup sweep moved from run_job() into tick(), after the ThreadPoolExecutor has joined every future. At that point no in-flight sessions from this tick can exist, so the sweep is always safe regardless of cron.max_parallel_jobs.

Related Issue

Related: #11202

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests
• ♻️ Refactor
• 🎯 New skill

Changes Made

• tools/mcp_tool.py
  • New module-level _orphan_stdio_pids: set separated from active _stdio_pids (active set is a dict of pid → server name, per upstream).
  • _run_stdio wrapped in try/finally; on exit any still-alive PID is migrated to the orphan set (liveness probed with signal 0). Keeps upstream stderr log routing (errlog=) in the same block.
  • _kill_orphaned_mcp_children gains include_active: bool = False; shutdown call site updated to include_active=True.
• cron/scheduler.py
  • Cleanup hook removed from run_job()'s finally:.
  • New cleanup hook at the end of tick(), after _results = [f.result() for f in _futures] but before the lock-release finally:, so it runs exactly once per tick and never during a sibling job.
• tests/tools/test_mcp_stability.py — TestStdioPidTracking targets _orphan_stdio_pids for the default (orphan-only) kill path, matching the v2 contract.

How to Test

Reproduction (before fix):

1. Start gateway with at least one stdio MCP server configured (e.g. mempalace).
2. Enable a recurring cron job whose agent invokes an MCP tool.
3. ps -eo pid,lstart,cmd | grep mempalace.mcp_server before and after a few ticks — each tick that suffers a cancellation leaves a surviving session-leader.

After fix:

1. Same setup. Run HERMES_CRON_MAX_PARALLEL=unbounded (default).
2. Trigger several ticks with overlapping jobs.
3. ps shows at most the actively connected MCP servers; previous ticks' orphans are reaped in the next tick's post-executor sweep. The currently active sessions of concurrent siblings are never touched.

Live-run observation on a Fedora/Asahi gateway running 7 cron jobs: before the patch, 6 orphan mempalace.mcp_server processes had accumulated across 24h (~570 MB RSS). After rebasing to this revision and restarting the gateway, the orphan count stays at 0 across the cron schedule.

Checklist

• My code follows the project's coding style
• I have commented my code where complex logic isn't self-evident
• I have added/updated tests (TestStdioPidTracking for orphan-only reaping; full suite not run in CI from this environment)
• All existing tests pass locally (syntax + import smoke on patched modules; TestStdioPidTracking run for this change)